2024-03-25T16:16:59.013 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:16:59.013 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621268 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --verbose --owner scheduled_amathuri@teuthology -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621268/orig.config.yaml 2024-03-25T16:16:59.058 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:16:59.139 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621268 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621268' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi122.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGd7eAzSBHm5d/7BTcOsLLUMmH7tiZ52/vuXEdLczaKP5XPiYpF/sMjLY0JGojUMgBC7U39UxBfDgCB94CgY6dc= smithi204.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEF5xawqKZ/fdvHRzou19KJf/TWLawveGfsRh6QIGO3NEYljtxhlVih68qNJP/Tfjrq49W/+x4/ZkKNnGSgRMmA= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:16:59.140 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:16:59.140 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:16:59.141 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:16:59.142 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:16:59.166 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:16:59.166 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:16:59.166 INFO:teuthology.packaging:ref: None 2024-03-25T16:16:59.166 INFO:teuthology.packaging:tag: None 2024-03-25T16:16:59.166 INFO:teuthology.packaging:branch: main 2024-03-25T16:16:59.166 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:16:59.167 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:16:59.395 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:16:59.397 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:16:59.404 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:16:59.404 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:16:59.442 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:16:59.461 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:16:59.467 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:16:59.494 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621268', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:11:41.766133', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGd7eAzSBHm5d/7BTcOsLLUMmH7tiZ52/vuXEdLczaKP5XPiYpF/sMjLY0JGojUMgBC7U39UxBfDgCB94CgY6dc='} 2024-03-25T16:16:59.516 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621268', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:11:41.767667', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEF5xawqKZ/fdvHRzou19KJf/TWLawveGfsRh6QIGO3NEYljtxhlVih68qNJP/Tfjrq49W/+x4/ZkKNnGSgRMmA='} 2024-03-25T16:16:59.516 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:16:59.522 INFO:teuthology.task.internal:roles: ubuntu@smithi122.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:16:59.522 INFO:teuthology.task.internal:roles: ubuntu@smithi204.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:16:59.522 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:16:59.651 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb9c7a7a0d0>, signals=[15]) 2024-03-25T16:16:59.651 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:16:59.657 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:16:59.657 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi122.front.sepia.ceph.com 2024-03-25T16:16:59.660 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:16:59.754 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi204.front.sepia.ceph.com 2024-03-25T16:16:59.756 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:16:59.849 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:16:59.855 DEBUG:teuthology.orchestra.run.smithi122:> uname -m 2024-03-25T16:16:59.873 INFO:teuthology.orchestra.run.smithi122.stdout:x86_64 2024-03-25T16:16:59.873 DEBUG:teuthology.orchestra.run.smithi122:> cat /etc/os-release 2024-03-25T16:16:59.929 INFO:teuthology.orchestra.run.smithi122.stdout:NAME="CentOS Stream" 2024-03-25T16:16:59.930 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION="9" 2024-03-25T16:16:59.930 INFO:teuthology.orchestra.run.smithi122.stdout:ID="centos" 2024-03-25T16:16:59.930 INFO:teuthology.orchestra.run.smithi122.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:16:59.930 INFO:teuthology.orchestra.run.smithi122.stdout:VERSION_ID="9" 2024-03-25T16:16:59.930 INFO:teuthology.orchestra.run.smithi122.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:16:59.930 INFO:teuthology.orchestra.run.smithi122.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:16:59.930 INFO:teuthology.orchestra.run.smithi122.stdout:ANSI_COLOR="0;31" 2024-03-25T16:16:59.930 INFO:teuthology.orchestra.run.smithi122.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:16:59.930 INFO:teuthology.orchestra.run.smithi122.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:16:59.930 INFO:teuthology.orchestra.run.smithi122.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:16:59.930 INFO:teuthology.orchestra.run.smithi122.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:16:59.930 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:16:59.930 INFO:teuthology.orchestra.run.smithi122.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:16:59.931 INFO:teuthology.lock.ops:Updating smithi122.front.sepia.ceph.com on lock server 2024-03-25T16:16:59.955 DEBUG:teuthology.orchestra.run.smithi204:> uname -m 2024-03-25T16:16:59.972 INFO:teuthology.orchestra.run.smithi204.stdout:x86_64 2024-03-25T16:16:59.973 DEBUG:teuthology.orchestra.run.smithi204:> cat /etc/os-release 2024-03-25T16:17:00.029 INFO:teuthology.orchestra.run.smithi204.stdout:NAME="CentOS Stream" 2024-03-25T16:17:00.029 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION="9" 2024-03-25T16:17:00.029 INFO:teuthology.orchestra.run.smithi204.stdout:ID="centos" 2024-03-25T16:17:00.029 INFO:teuthology.orchestra.run.smithi204.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:17:00.029 INFO:teuthology.orchestra.run.smithi204.stdout:VERSION_ID="9" 2024-03-25T16:17:00.029 INFO:teuthology.orchestra.run.smithi204.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:17:00.029 INFO:teuthology.orchestra.run.smithi204.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:17:00.029 INFO:teuthology.orchestra.run.smithi204.stdout:ANSI_COLOR="0;31" 2024-03-25T16:17:00.029 INFO:teuthology.orchestra.run.smithi204.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:17:00.029 INFO:teuthology.orchestra.run.smithi204.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:17:00.030 INFO:teuthology.orchestra.run.smithi204.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:17:00.030 INFO:teuthology.orchestra.run.smithi204.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:17:00.030 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:17:00.030 INFO:teuthology.orchestra.run.smithi204.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:17:00.030 INFO:teuthology.lock.ops:Updating smithi204.front.sepia.ceph.com on lock server 2024-03-25T16:17:00.047 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:17:00.056 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:17:00.075 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:17:00.076 DEBUG:teuthology.orchestra.run.smithi122:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:17:00.078 DEBUG:teuthology.orchestra.run.smithi204:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:17:00.093 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:17:00.101 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:17:00.101 DEBUG:teuthology.orchestra.run.smithi122:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:17:00.134 DEBUG:teuthology.orchestra.run.smithi204:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:17:00.246 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:17:00.298 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:17:00.313 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:17:00.313 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:17:00.313 DEBUG:teuthology.orchestra.run.smithi122:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:17:00.314 DEBUG:teuthology.orchestra.run.smithi204:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:17:00.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:17:00.330 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-03-25T16:17:00.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:17:00.331 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-03-25T16:17:00.387 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:17:00.387 INFO:teuthology.task.kernel:Running kernel on smithi204: 5.14.0-361.el9.x86_64 2024-03-25T16:17:00.387 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum install -y kernel 2024-03-25T16:17:00.388 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:17:00.388 INFO:teuthology.task.kernel:Running kernel on smithi122: 5.14.0-361.el9.x86_64 2024-03-25T16:17:00.388 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum install -y kernel 2024-03-25T16:17:02.933 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - AppStream 7.6 MB/s | 6.7 MB 00:00 2024-03-25T16:17:02.990 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - AppStream 8.0 MB/s | 6.7 MB 00:00 2024-03-25T16:17:04.534 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - BaseOS 4.5 MB/s | 2.0 MB 00:00 2024-03-25T16:17:04.615 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-03-25T16:17:07.499 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 3.0 MB/s | 8.0 MB 00:02 2024-03-25T16:17:09.467 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-25T16:17:13.401 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-25T16:17:14.113 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 909 kB/s | 8.0 MB 00:09 2024-03-25T16:17:14.827 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 52 kB/s | 16 kB 00:00 2024-03-25T16:17:16.771 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-25T16:17:21.771 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-25T16:17:23.894 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:17:23.894 INFO:teuthology.orchestra.run.smithi122.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:17:24.010 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:17:24.016 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:17:24.016 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-03-25T16:17:24.016 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:17:24.016 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:17:24.016 INFO:teuthology.orchestra.run.smithi122.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:17:24.016 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-25T16:17:24.016 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:17:24.016 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:17:24.016 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:17:24.016 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:17:24.016 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:17:24.016 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:17:24.017 INFO:teuthology.orchestra.run.smithi122.stdout:Install 4 Packages 2024-03-25T16:17:24.017 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:17:24.017 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 76 M 2024-03-25T16:17:24.017 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 123 M 2024-03-25T16:17:24.017 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:17:24.289 INFO:teuthology.orchestra.run.smithi122.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 460 kB/s | 44 kB 00:00 2024-03-25T16:17:25.323 INFO:teuthology.orchestra.run.smithi122.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 13 MB/s | 15 MB 00:01 2024-03-25T16:17:26.198 INFO:teuthology.orchestra.run.smithi122.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 17 MB/s | 33 MB 00:02 2024-03-25T16:17:26.473 INFO:teuthology.orchestra.run.smithi122.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 13 MB/s | 27 MB 00:02 2024-03-25T16:17:26.474 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:26.474 INFO:teuthology.orchestra.run.smithi122.stdout:Total 31 MB/s | 76 MB 00:02 2024-03-25T16:17:27.138 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:17:27.508 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:17:27.508 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:17:28.515 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:17:28.516 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:17:31.035 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:17:31.712 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:17:31.809 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:17:34.554 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:17:34.691 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:17:36.229 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 916 kB/s | 19 MB 00:21 2024-03-25T16:17:39.385 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:17:39.500 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:40.646 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 7.1 MB/s | 6.1 MB 00:00 2024-03-25T16:17:42.863 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-25T16:17:44.263 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:44.976 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:01 2024-03-25T16:17:49.915 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-25T16:17:52.055 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:17:52.056 INFO:teuthology.orchestra.run.smithi204.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:17:52.143 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:17:52.152 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:17:52.153 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-03-25T16:17:52.153 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:17:52.153 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:17:52.153 INFO:teuthology.orchestra.run.smithi204.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:17:52.153 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-25T16:17:52.153 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:17:52.153 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:17:52.153 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:17:52.153 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:17:52.153 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:17:52.153 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:17:52.153 INFO:teuthology.orchestra.run.smithi204.stdout:Install 4 Packages 2024-03-25T16:17:52.153 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:17:52.154 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 76 M 2024-03-25T16:17:52.154 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 123 M 2024-03-25T16:17:52.154 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:17:52.333 INFO:teuthology.orchestra.run.smithi204.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 382 kB/s | 44 kB 00:00 2024-03-25T16:17:53.508 INFO:teuthology.orchestra.run.smithi204.stdout:(2/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 26 MB/s | 33 MB 00:01 2024-03-25T16:17:54.000 INFO:teuthology.orchestra.run.smithi204.stdout:(3/4): kernel-core-5.14.0-430.el9.x86_64.rpm 8.4 MB/s | 15 MB 00:01 2024-03-25T16:17:54.417 INFO:teuthology.orchestra.run.smithi204.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 13 MB/s | 27 MB 00:02 2024-03-25T16:17:54.417 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:54.417 INFO:teuthology.orchestra.run.smithi204.stdout:Total 34 MB/s | 76 MB 00:02 2024-03-25T16:17:55.083 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:17:55.443 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:17:55.444 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:17:56.437 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:17:56.437 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:17:58.929 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:17:59.548 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:17:59.626 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:18:02.368 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:18:02.506 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:18:07.224 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:18:07.264 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:18:12.010 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:18:13.660 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:18:13.660 INFO:teuthology.orchestra.run.smithi122.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:18:13.660 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:18:13.704 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:18:14.110 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:18:14.110 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:18:14.110 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:18:14.110 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:18:14.553 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:18:14.554 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:18:14.554 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:18:14.554 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:18:14.554 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:18:14.554 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:18:14.554 INFO:teuthology.orchestra.run.smithi122.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:18:14.554 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:18:14.555 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:18:14.897 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:18:14.997 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:18:14.997 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:18:14.997 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:18:14.997 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:18:14.997 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:18:14.997 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:18:14.997 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi122.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:18:14.998 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:18:15.076 INFO:teuthology.orchestra.run.smithi122.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:18:15.077 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:18:15.078 DEBUG:teuthology.orchestra.run.smithi122:> sudo rpm -qi grub2-tools 2024-03-25T16:18:15.111 INFO:teuthology.orchestra.run.smithi122.stdout:Name : grub2-tools 2024-03-25T16:18:15.111 INFO:teuthology.orchestra.run.smithi122.stdout:Epoch : 1 2024-03-25T16:18:15.111 INFO:teuthology.orchestra.run.smithi122.stdout:Version : 2.06 2024-03-25T16:18:15.111 INFO:teuthology.orchestra.run.smithi122.stdout:Release : 68.el9 2024-03-25T16:18:15.111 INFO:teuthology.orchestra.run.smithi122.stdout:Architecture: x86_64 2024-03-25T16:18:15.111 INFO:teuthology.orchestra.run.smithi122.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:18:15.111 INFO:teuthology.orchestra.run.smithi122.stdout:Group : Unspecified 2024-03-25T16:18:15.111 INFO:teuthology.orchestra.run.smithi122.stdout:Size : 8264254 2024-03-25T16:18:15.111 INFO:teuthology.orchestra.run.smithi122.stdout:License : GPLv3+ 2024-03-25T16:18:15.111 INFO:teuthology.orchestra.run.smithi122.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:18:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:18:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:18:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:18:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:Packager : builder@centos.org 2024-03-25T16:18:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:Vendor : CentOS 2024-03-25T16:18:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:18:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:Summary : Support tools for GRUB. 2024-03-25T16:18:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:Description : 2024-03-25T16:18:15.112 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:18:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:18:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:18:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:18:15.112 INFO:teuthology.orchestra.run.smithi122.stdout:hardware devices. 2024-03-25T16:18:15.113 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:18:15.113 INFO:teuthology.orchestra.run.smithi122.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:18:15.113 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:18:15.114 INFO:teuthology.task.kernel:Updating grub on smithi122 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:18:15.114 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:18:15.676 INFO:teuthology.orchestra.run.smithi122.stderr:Generating grub configuration file ... 2024-03-25T16:18:17.067 INFO:teuthology.orchestra.run.smithi122.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:18:17.097 INFO:teuthology.orchestra.run.smithi122.stderr:done 2024-03-25T16:18:17.099 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-03-25T16:18:17.114 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.QJ46OorlZZ 2024-03-25T16:18:17.114 DEBUG:teuthology.orchestra.run.smithi122:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QJ46OorlZZ 2024-03-25T16:18:17.182 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.QJ46OorlZZ 2024-03-25T16:18:17.337 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.QJ46OorlZZ is 6KB 2024-03-25T16:18:17.385 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.QJ46OorlZZ 2024-03-25T16:18:17.399 DEBUG:teuthology.orchestra.run.smithi122:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:18:17.467 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:18:17.467 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:18:17.467 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:18:17.467 INFO:teuthology.orchestra.run.smithi122.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:18:17.469 DEBUG:teuthology.orchestra.run.smithi122:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:18:17.627 DEBUG:teuthology.orchestra.run.smithi122:> sudo shutdown -r now 2024-03-25T16:18:41.215 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:18:41.215 INFO:teuthology.orchestra.run.smithi204.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:18:41.215 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:18:41.258 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:18:41.620 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:18:41.620 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:18:41.620 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:18:41.620 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:18:42.009 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:18:42.010 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:18:42.010 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:18:42.010 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:18:42.010 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:18:42.010 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:18:42.010 INFO:teuthology.orchestra.run.smithi204.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:18:42.010 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:18:42.010 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:18:42.332 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:18:42.434 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:18:42.434 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:18:42.434 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:18:42.434 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:18:42.434 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:18:42.434 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:18:42.434 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi204.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:18:42.435 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:18:42.515 INFO:teuthology.orchestra.run.smithi204.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:18:42.515 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:18:42.516 DEBUG:teuthology.orchestra.run.smithi204:> sudo rpm -qi grub2-tools 2024-03-25T16:18:42.555 INFO:teuthology.orchestra.run.smithi204.stdout:Name : grub2-tools 2024-03-25T16:18:42.555 INFO:teuthology.orchestra.run.smithi204.stdout:Epoch : 1 2024-03-25T16:18:42.555 INFO:teuthology.orchestra.run.smithi204.stdout:Version : 2.06 2024-03-25T16:18:42.556 INFO:teuthology.orchestra.run.smithi204.stdout:Release : 68.el9 2024-03-25T16:18:42.556 INFO:teuthology.orchestra.run.smithi204.stdout:Architecture: x86_64 2024-03-25T16:18:42.556 INFO:teuthology.orchestra.run.smithi204.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:18:42.556 INFO:teuthology.orchestra.run.smithi204.stdout:Group : Unspecified 2024-03-25T16:18:42.556 INFO:teuthology.orchestra.run.smithi204.stdout:Size : 8264254 2024-03-25T16:18:42.556 INFO:teuthology.orchestra.run.smithi204.stdout:License : GPLv3+ 2024-03-25T16:18:42.556 INFO:teuthology.orchestra.run.smithi204.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:18:42.556 INFO:teuthology.orchestra.run.smithi204.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:18:42.556 INFO:teuthology.orchestra.run.smithi204.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:18:42.556 INFO:teuthology.orchestra.run.smithi204.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:18:42.556 INFO:teuthology.orchestra.run.smithi204.stdout:Packager : builder@centos.org 2024-03-25T16:18:42.556 INFO:teuthology.orchestra.run.smithi204.stdout:Vendor : CentOS 2024-03-25T16:18:42.556 INFO:teuthology.orchestra.run.smithi204.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:18:42.556 INFO:teuthology.orchestra.run.smithi204.stdout:Summary : Support tools for GRUB. 2024-03-25T16:18:42.557 INFO:teuthology.orchestra.run.smithi204.stdout:Description : 2024-03-25T16:18:42.557 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:18:42.557 INFO:teuthology.orchestra.run.smithi204.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:18:42.557 INFO:teuthology.orchestra.run.smithi204.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:18:42.557 INFO:teuthology.orchestra.run.smithi204.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:18:42.557 INFO:teuthology.orchestra.run.smithi204.stdout:hardware devices. 2024-03-25T16:18:42.557 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:18:42.557 INFO:teuthology.orchestra.run.smithi204.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:18:42.558 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:18:42.558 INFO:teuthology.task.kernel:Updating grub on smithi204 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:18:42.558 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:18:43.162 INFO:teuthology.orchestra.run.smithi204.stderr:Generating grub configuration file ... 2024-03-25T16:18:44.513 INFO:teuthology.orchestra.run.smithi204.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:18:44.562 INFO:teuthology.orchestra.run.smithi204.stderr:done 2024-03-25T16:18:44.564 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-03-25T16:18:44.579 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.4SxBLEoE6S 2024-03-25T16:18:44.579 DEBUG:teuthology.orchestra.run.smithi204:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.4SxBLEoE6S 2024-03-25T16:18:44.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.4SxBLEoE6S 2024-03-25T16:18:44.783 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.4SxBLEoE6S is 6KB 2024-03-25T16:18:44.830 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.4SxBLEoE6S 2024-03-25T16:18:44.844 DEBUG:teuthology.orchestra.run.smithi204:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:18:44.909 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:18:44.909 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:18:44.909 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:18:44.909 INFO:teuthology.orchestra.run.smithi204.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:18:44.911 DEBUG:teuthology.orchestra.run.smithi204:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:18:45.065 DEBUG:teuthology.orchestra.run.smithi204:> sudo shutdown -r now 2024-03-25T16:18:47.630 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:18:47.631 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-25T16:18:47.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:19:15.094 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:19:15.095 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-25T16:19:15.096 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:19:33.535 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-03-25T16:19:42.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-25T16:19:42.545 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:19:45.598 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.204 2024-03-25T16:19:47.634 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:19:56.642 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-25T16:19:56.643 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:19:57.306 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-03-25T16:19:57.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-25T16:19:57.600 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:19:57.739 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-25T16:19:57.739 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:19:57.739 DEBUG:teuthology.orchestra.run.smithi122:> uname -r 2024-03-25T16:19:57.753 INFO:teuthology.orchestra.run.smithi122.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:19:57.753 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:19:57.753 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:19:57.753 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:19:58.755 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:19:58.755 DEBUG:teuthology.orchestra.run.smithi122:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:19:58.844 INFO:teuthology.orchestra.run.smithi122.stdout:ttyS1 2024-03-25T16:19:58.880 DEBUG:teuthology.parallel:result is None 2024-03-25T16:20:57.602 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:21:12.603 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-25T16:21:12.605 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:21:13.045 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-03-25T16:21:13.440 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-25T16:21:13.440 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:21:13.440 DEBUG:teuthology.orchestra.run.smithi204:> uname -r 2024-03-25T16:21:13.455 INFO:teuthology.orchestra.run.smithi204.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:21:13.455 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:21:13.455 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:21:13.456 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:21:14.457 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:21:14.457 DEBUG:teuthology.orchestra.run.smithi204:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:21:14.535 INFO:teuthology.orchestra.run.smithi204.stdout:ttyS1 2024-03-25T16:21:14.570 DEBUG:teuthology.parallel:result is None 2024-03-25T16:21:14.571 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:21:14.578 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:21:14.579 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:21:14.582 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:21:14.600 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:21:14.640 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:21:14.671 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:21:14.671 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:21:14.674 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:21:14.718 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:21:14.724 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:21:14.725 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:21:14.755 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:21:14.799 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:21:14.807 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:21:14.812 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:21:14.822 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:21:14.824 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:21:14.831 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:21:14.831 DEBUG:teuthology.orchestra.run.smithi122:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:21:14.833 DEBUG:teuthology.orchestra.run.smithi204:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:21:14.884 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:21:14.894 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:21:14.894 DEBUG:teuthology.orchestra.run.smithi122:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:21:14.906 DEBUG:teuthology.orchestra.run.smithi204:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:21:14.940 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:21:15.007 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:21:15.056 DEBUG:teuthology.orchestra.run.smithi122:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:21:15.142 DEBUG:teuthology.orchestra.run.smithi122:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:21:15.188 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T16:21:15.188 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:21:15.258 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:21:15.298 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:21:15.351 DEBUG:teuthology.orchestra.run.smithi204:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:21:15.431 DEBUG:teuthology.orchestra.run.smithi204:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:21:15.482 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T16:21:15.482 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:21:15.550 DEBUG:teuthology.orchestra.run.smithi122:> sudo service rsyslog restart 2024-03-25T16:21:15.553 DEBUG:teuthology.orchestra.run.smithi204:> sudo service rsyslog restart 2024-03-25T16:21:15.609 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:21:15.642 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:21:16.083 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:21:16.094 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:21:16.094 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:21:16.143 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:21:16.175 DEBUG:teuthology.orchestra.run.smithi122:> sudo service auditd rotate 2024-03-25T16:21:16.237 INFO:teuthology.orchestra.run.smithi122.stdout:Rotating logs: 2024-03-25T16:21:16.239 DEBUG:teuthology.orchestra.run.smithi204:> sudo service auditd rotate 2024-03-25T16:21:16.306 INFO:teuthology.orchestra.run.smithi204.stdout:Rotating logs: 2024-03-25T16:21:16.308 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:21:16.309 DEBUG:teuthology.orchestra.run.smithi122:> /usr/sbin/getenforce 2024-03-25T16:21:16.331 INFO:teuthology.orchestra.run.smithi122.stdout:Permissive 2024-03-25T16:21:16.331 DEBUG:teuthology.orchestra.run.smithi204:> /usr/sbin/getenforce 2024-03-25T16:21:16.362 INFO:teuthology.orchestra.run.smithi204.stdout:Permissive 2024-03-25T16:21:16.362 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi122.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi204.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:21:16.362 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:21:16.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:21:16.400 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:21:16.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:21:16.430 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:21:16.430 DEBUG:teuthology.orchestra.run.smithi122:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:21:16.477 DEBUG:teuthology.orchestra.run.smithi204:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:21:16.513 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:21:16.523 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:21:16.541 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:21:16.543 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi122.front.sepia.ceph.com,smithi204.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:27:49.909 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi122.front.sepia.ceph.com'), Remote(name='ubuntu@smithi204.front.sepia.ceph.com')] 2024-03-25T16:27:49.911 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-25T16:27:49.912 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi122.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:27:50.001 DEBUG:teuthology.orchestra.run.smithi122:> true 2024-03-25T16:27:50.084 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi122.front.sepia.ceph.com' 2024-03-25T16:27:50.085 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-25T16:27:50.085 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi204.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:27:50.175 DEBUG:teuthology.orchestra.run.smithi204:> true 2024-03-25T16:27:50.259 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi204.front.sepia.ceph.com' 2024-03-25T16:27:50.259 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:27:50.270 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:27:50.270 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:27:50.271 DEBUG:teuthology.orchestra.run.smithi122:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:27:50.273 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:27:50.273 DEBUG:teuthology.orchestra.run.smithi204:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:27:50.301 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:27:50.317 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:27:50.332 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:27:50.346 INFO:teuthology.orchestra.run.smithi122.stderr:sudo: ntpd: command not found 2024-03-25T16:27:50.350 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:27:50.360 INFO:teuthology.orchestra.run.smithi122.stdout:506 Cannot talk to daemon 2024-03-25T16:27:50.376 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:27:50.380 INFO:teuthology.orchestra.run.smithi204.stderr:sudo: ntpd: command not found 2024-03-25T16:27:50.393 INFO:teuthology.orchestra.run.smithi122.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:27:50.393 INFO:teuthology.orchestra.run.smithi204.stdout:506 Cannot talk to daemon 2024-03-25T16:27:50.410 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:27:50.426 INFO:teuthology.orchestra.run.smithi204.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:27:50.443 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:27:50.445 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:27:50.446 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-03-25T16:27:50.446 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:27:50.446 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:27:50.446 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:27:50.446 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:27:50.478 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:27:50.481 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:27:50.481 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-03-25T16:27:50.481 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:27:50.481 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:27:50.481 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:27:50.481 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:27:50.482 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:27:50.490 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:27:50.490 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:27:50.491 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:27:50.491 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:27:50.496 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:27:50.496 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:27:50.497 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:27:50.497 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:27:50.499 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:27:50.500 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:27:50.659 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:27:50.659 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:27:50.713 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:27:50.713 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:27:51.060 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:27:51.060 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T16:27:51.060 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:27:51.062 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:27:51.062 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T16:27:51.062 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:27:51.091 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:27:51.091 DEBUG:teuthology.orchestra.run.smithi122:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:27:51.092 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:27:51.093 DEBUG:teuthology.orchestra.run.smithi204:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:27:51.160 DEBUG:teuthology.orchestra.run.smithi122:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:27:51.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:27:51.241 DEBUG:teuthology.orchestra.run.smithi122:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:27:51.242 DEBUG:teuthology.orchestra.run.smithi204:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:27:51.307 INFO:teuthology.orchestra.run.smithi122.stdout:check_obsoletes = 1 2024-03-25T16:27:51.309 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum clean all 2024-03-25T16:27:51.310 INFO:teuthology.orchestra.run.smithi204.stdout:check_obsoletes = 1 2024-03-25T16:27:51.310 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum clean all 2024-03-25T16:27:51.546 INFO:teuthology.orchestra.run.smithi122.stdout:74 files removed 2024-03-25T16:27:51.564 INFO:teuthology.orchestra.run.smithi204.stdout:74 files removed 2024-03-25T16:27:51.572 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-radosgw 2024-03-25T16:27:51.591 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-radosgw 2024-03-25T16:27:53.095 INFO:teuthology.orchestra.run.smithi204.stdout:ceph packages for x86_64 68 kB/s | 82 kB 00:01 2024-03-25T16:27:53.913 INFO:teuthology.orchestra.run.smithi204.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T16:27:54.739 INFO:teuthology.orchestra.run.smithi204.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T16:27:55.207 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-25T16:27:56.757 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2024-03-25T16:27:59.249 INFO:teuthology.orchestra.run.smithi122.stdout:ceph packages for x86_64 11 kB/s | 82 kB 00:07 2024-03-25T16:27:59.615 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - BaseOS 3.2 MB/s | 8.0 MB 00:02 2024-03-25T16:28:00.174 INFO:teuthology.orchestra.run.smithi122.stdout:ceph noarch packages 11 kB/s | 10 kB 00:00 2024-03-25T16:28:01.092 INFO:teuthology.orchestra.run.smithi122.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-03-25T16:28:01.810 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - AppStream 9.9 MB/s | 6.7 MB 00:00 2024-03-25T16:28:02.249 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-25T16:28:03.510 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS-9 - BaseOS 3.5 MB/s | 2.0 MB 00:00 2024-03-25T16:28:04.268 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-25T16:28:06.244 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - AppStream 20 MB/s | 19 MB 00:00 2024-03-25T16:28:10.545 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.1 MB 00:00 2024-03-25T16:28:11.896 INFO:teuthology.orchestra.run.smithi122.stdout:CentOS Stream 9 - Extras packages 87 kB/s | 16 kB 00:00 2024-03-25T16:28:13.524 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - CRB 808 kB/s | 6.1 MB 00:07 2024-03-25T16:28:13.589 INFO:teuthology.orchestra.run.smithi122.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-03-25T16:28:15.375 INFO:teuthology.orchestra.run.smithi204.stdout:CentOS Stream 9 - Extras packages 22 kB/s | 16 kB 00:00 2024-03-25T16:28:17.671 INFO:teuthology.orchestra.run.smithi204.stdout:Extra Packages for Enterprise Linux 9.5 MB/s | 21 MB 00:02 2024-03-25T16:28:18.539 INFO:teuthology.orchestra.run.smithi122.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-25T16:28:21.159 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:28:21.174 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:28:21.175 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-25T16:28:21.175 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:28:21.175 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:28:21.175 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:28:21.175 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrading: 2024-03-25T16:28:21.175 INFO:teuthology.orchestra.run.smithi122.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:28:21.175 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:28:21.175 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-25T16:28:21.175 INFO:teuthology.orchestra.run.smithi122.stdout: boost-program-options 2024-03-25T16:28:21.175 INFO:teuthology.orchestra.run.smithi122.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:28:21.175 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:28:21.175 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:28:21.176 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:28:21.176 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:28:21.176 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:28:21.176 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:28:21.176 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:28:21.176 INFO:teuthology.orchestra.run.smithi122.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:28:21.176 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:28:21.176 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:28:21.176 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:28:21.176 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:28:21.176 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:28:21.176 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:28:21.176 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:28:21.176 INFO:teuthology.orchestra.run.smithi122.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:28:21.176 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse 2024-03-25T16:28:21.177 INFO:teuthology.orchestra.run.smithi122.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:28:21.177 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:28:21.177 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:28:21.177 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:28:21.177 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:28:21.177 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:28:21.177 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:28:21.177 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:28:21.177 INFO:teuthology.orchestra.run.smithi122.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:28:21.177 INFO:teuthology.orchestra.run.smithi122.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:28:21.177 INFO:teuthology.orchestra.run.smithi122.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:28:21.177 INFO:teuthology.orchestra.run.smithi122.stdout:Installing weak dependencies: 2024-03-25T16:28:21.177 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:28:21.177 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:28:21.177 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:28:21.178 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:28:21.178 INFO:teuthology.orchestra.run.smithi122.stdout:Install 30 Packages 2024-03-25T16:28:21.178 INFO:teuthology.orchestra.run.smithi122.stdout:Upgrade 2 Packages 2024-03-25T16:28:21.178 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:28:21.180 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 58 M 2024-03-25T16:28:21.180 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:28:22.560 INFO:teuthology.orchestra.run.smithi204.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-25T16:28:24.988 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:28:25.000 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:28:25.000 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-25T16:28:25.000 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:28:25.000 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:28:25.000 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:28:25.000 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrading: 2024-03-25T16:28:25.000 INFO:teuthology.orchestra.run.smithi204.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:28:25.000 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:28:25.001 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-25T16:28:25.001 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options 2024-03-25T16:28:25.001 INFO:teuthology.orchestra.run.smithi204.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:28:25.001 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:28:25.001 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:28:25.001 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:28:25.001 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:28:25.001 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:28:25.001 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:28:25.001 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:28:25.001 INFO:teuthology.orchestra.run.smithi204.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:28:25.001 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:28:25.001 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:28:25.001 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:28:25.002 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:28:25.002 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:28:25.002 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:28:25.002 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:28:25.002 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:28:25.002 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse 2024-03-25T16:28:25.002 INFO:teuthology.orchestra.run.smithi204.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:28:25.002 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:28:25.002 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:28:25.002 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:28:25.002 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:28:25.002 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:28:25.002 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:28:25.002 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:28:25.002 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:28:25.003 INFO:teuthology.orchestra.run.smithi204.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:28:25.003 INFO:teuthology.orchestra.run.smithi204.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:28:25.003 INFO:teuthology.orchestra.run.smithi204.stdout:Installing weak dependencies: 2024-03-25T16:28:25.003 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:28:25.003 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:28:25.003 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:28:25.003 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:28:25.003 INFO:teuthology.orchestra.run.smithi204.stdout:Install 30 Packages 2024-03-25T16:28:25.003 INFO:teuthology.orchestra.run.smithi204.stdout:Upgrade 2 Packages 2024-03-25T16:28:25.003 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:28:25.004 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 58 M 2024-03-25T16:28:25.004 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:28:35.931 INFO:teuthology.orchestra.run.smithi122.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 388 kB/s | 5.3 MB 00:14 2024-03-25T16:28:36.031 INFO:teuthology.orchestra.run.smithi122.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T16:28:37.548 INFO:teuthology.orchestra.run.smithi122.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 489 kB/s | 742 kB 00:01 2024-03-25T16:28:38.489 INFO:teuthology.orchestra.run.smithi122.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 529 kB/s | 498 kB 00:00 2024-03-25T16:28:45.676 INFO:teuthology.orchestra.run.smithi204.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 488 kB/s | 9.7 MB 00:20 2024-03-25T16:28:45.743 INFO:teuthology.orchestra.run.smithi204.stdout:(2/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 267 kB/s | 5.3 MB 00:20 2024-03-25T16:28:45.785 INFO:teuthology.orchestra.run.smithi204.stdout:(3/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 232 kB/s | 25 kB 00:00 2024-03-25T16:28:47.135 INFO:teuthology.orchestra.run.smithi204.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 369 kB/s | 498 kB 00:01 2024-03-25T16:28:48.343 INFO:teuthology.orchestra.run.smithi204.stdout:(5/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 285 kB/s | 742 kB 00:02 2024-03-25T16:28:48.447 INFO:teuthology.orchestra.run.smithi122.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 375 kB/s | 9.7 MB 00:26 2024-03-25T16:28:48.560 INFO:teuthology.orchestra.run.smithi204.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 208 kB/s | 45 kB 00:00 2024-03-25T16:28:48.673 INFO:teuthology.orchestra.run.smithi122.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 201 kB/s | 45 kB 00:00 2024-03-25T16:28:49.352 INFO:teuthology.orchestra.run.smithi204.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 166 kB/s | 131 kB 00:00 2024-03-25T16:28:49.398 INFO:teuthology.orchestra.run.smithi122.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 181 kB/s | 131 kB 00:00 2024-03-25T16:28:49.853 INFO:teuthology.orchestra.run.smithi204.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 324 kB/s | 162 kB 00:00 2024-03-25T16:28:50.058 INFO:teuthology.orchestra.run.smithi122.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 246 kB/s | 162 kB 00:00 2024-03-25T16:28:50.736 INFO:teuthology.orchestra.run.smithi204.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 363 kB/s | 320 kB 00:00 2024-03-25T16:28:51.124 INFO:teuthology.orchestra.run.smithi122.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 300 kB/s | 320 kB 00:01 2024-03-25T16:28:51.904 INFO:teuthology.orchestra.run.smithi204.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 255 kB/s | 298 kB 00:01 2024-03-25T16:28:52.066 INFO:teuthology.orchestra.run.smithi122.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 316 kB/s | 298 kB 00:00 2024-03-25T16:28:52.741 INFO:teuthology.orchestra.run.smithi122.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 147 kB/s | 99 kB 00:00 2024-03-25T16:28:52.925 INFO:teuthology.orchestra.run.smithi122.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 577 kB/s | 106 kB 00:00 2024-03-25T16:28:52.983 INFO:teuthology.orchestra.run.smithi122.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-25T16:28:53.034 INFO:teuthology.orchestra.run.smithi122.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-25T16:28:53.159 INFO:teuthology.orchestra.run.smithi122.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 5.2 MB/s | 661 kB 00:00 2024-03-25T16:28:53.234 INFO:teuthology.orchestra.run.smithi122.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-25T16:28:53.245 INFO:teuthology.orchestra.run.smithi204.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 74 kB/s | 99 kB 00:01 2024-03-25T16:28:53.292 INFO:teuthology.orchestra.run.smithi122.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 708 kB/s | 41 kB 00:00 2024-03-25T16:28:53.395 INFO:teuthology.orchestra.run.smithi204.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 708 kB/s | 106 kB 00:00 2024-03-25T16:28:53.434 INFO:teuthology.orchestra.run.smithi122.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.5 MB/s | 216 kB 00:00 2024-03-25T16:28:53.454 INFO:teuthology.orchestra.run.smithi204.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 159 kB 00:00 2024-03-25T16:28:53.488 INFO:teuthology.orchestra.run.smithi204.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:28:53.579 INFO:teuthology.orchestra.run.smithi122.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 420 kB/s | 35 kB 00:00 2024-03-25T16:28:53.579 INFO:teuthology.orchestra.run.smithi204.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-25T16:28:53.597 INFO:teuthology.orchestra.run.smithi204.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:28:53.618 INFO:teuthology.orchestra.run.smithi122.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 446 kB/s | 44 kB 00:00 2024-03-25T16:28:53.630 INFO:teuthology.orchestra.run.smithi204.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:28:54.069 INFO:teuthology.orchestra.run.smithi122.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 687 kB/s | 309 kB 00:00 2024-03-25T16:28:54.222 INFO:teuthology.orchestra.run.smithi204.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 366 kB/s | 216 kB 00:00 2024-03-25T16:28:54.331 INFO:teuthology.orchestra.run.smithi204.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 322 kB/s | 35 kB 00:00 2024-03-25T16:28:54.427 INFO:teuthology.orchestra.run.smithi122.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-03-25T16:28:54.439 INFO:teuthology.orchestra.run.smithi204.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 411 kB/s | 44 kB 00:00 2024-03-25T16:28:54.511 INFO:teuthology.orchestra.run.smithi122.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 312 kB/s | 26 kB 00:00 2024-03-25T16:28:54.594 INFO:teuthology.orchestra.run.smithi122.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 585 kB/s | 49 kB 00:00 2024-03-25T16:28:54.686 INFO:teuthology.orchestra.run.smithi122.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 733 kB/s | 67 kB 00:00 2024-03-25T16:28:54.795 INFO:teuthology.orchestra.run.smithi122.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.6 MB/s | 838 kB 00:00 2024-03-25T16:28:54.831 INFO:teuthology.orchestra.run.smithi204.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 788 kB/s | 309 kB 00:00 2024-03-25T16:28:54.895 INFO:teuthology.orchestra.run.smithi122.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-25T16:28:55.098 INFO:teuthology.orchestra.run.smithi204.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-03-25T16:28:55.157 INFO:teuthology.orchestra.run.smithi204.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 446 kB/s | 26 kB 00:00 2024-03-25T16:28:55.216 INFO:teuthology.orchestra.run.smithi204.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 834 kB/s | 49 kB 00:00 2024-03-25T16:28:55.274 INFO:teuthology.orchestra.run.smithi204.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-25T16:28:55.375 INFO:teuthology.orchestra.run.smithi204.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-03-25T16:28:55.442 INFO:teuthology.orchestra.run.smithi204.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-25T16:28:55.551 INFO:teuthology.orchestra.run.smithi204.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-25T16:28:55.612 INFO:teuthology.orchestra.run.smithi122.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 2.2 MB/s | 1.6 MB 00:00 2024-03-25T16:28:56.043 INFO:teuthology.orchestra.run.smithi204.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 586 kB/s | 5.1 MB 00:08 2024-03-25T16:28:58.671 INFO:teuthology.orchestra.run.smithi122.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 259 kB/s | 5.1 MB 00:20 2024-03-25T16:29:02.567 INFO:teuthology.orchestra.run.smithi204.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 491 kB/s | 3.1 MB 00:06 2024-03-25T16:29:07.603 INFO:teuthology.orchestra.run.smithi122.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 288 kB/s | 3.4 MB 00:11 2024-03-25T16:29:09.370 INFO:teuthology.orchestra.run.smithi122.stdout:(31/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 428 kB/s | 20 MB 00:47 2024-03-25T16:29:10.466 INFO:teuthology.orchestra.run.smithi204.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 232 kB/s | 3.4 MB 00:14 2024-03-25T16:29:12.686 INFO:teuthology.orchestra.run.smithi122.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 229 kB/s | 3.1 MB 00:14 2024-03-25T16:29:12.690 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:12.690 INFO:teuthology.orchestra.run.smithi122.stdout:Total 1.1 MB/s | 58 MB 00:51 2024-03-25T16:29:12.724 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:29:12.762 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:29:12.762 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:29:13.225 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:29:13.225 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:29:14.677 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:29:14.769 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:29:14.799 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:29:14.825 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:29:14.848 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:29:14.864 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:29:15.013 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:29:15.070 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:29:15.387 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:29:15.451 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:29:15.534 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:29:15.621 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:29:15.644 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:29:15.698 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:29:15.727 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:29:15.762 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:29:15.830 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:29:15.855 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:29:15.918 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:29:15.944 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:29:16.051 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:29:16.079 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:29:16.119 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:29:16.152 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:29:16.181 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:29:16.214 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:29:16.523 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:29:16.552 INFO:teuthology.orchestra.run.smithi122.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:29:16.641 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:29:16.674 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:29:16.711 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:29:17.163 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:29:17.196 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:29:17.259 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:29:17.328 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:29:17.357 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:29:19.547 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:29:19.611 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:29:20.161 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:29:20.206 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:29:20.286 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:29:20.286 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:29:20.286 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:29:20.287 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:29:20.324 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:29:29.496 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:29:29.496 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /sys 2024-03-25T16:29:29.496 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /proc 2024-03-25T16:29:29.496 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /mnt 2024-03-25T16:29:29.496 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /var/tmp 2024-03-25T16:29:29.496 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /home 2024-03-25T16:29:29.496 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /root 2024-03-25T16:29:29.497 INFO:teuthology.orchestra.run.smithi122.stdout:skipping the directory /tmp 2024-03-25T16:29:29.497 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:29:30.360 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:29:30.384 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:29:30.384 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:29:30.384 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:29:30.384 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:29:30.384 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:29:30.384 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:29:30.385 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:29:30.461 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:29:30.461 INFO:teuthology.orchestra.run.smithi122.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:29:31.552 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:29:31.553 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:29:31.553 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:29:31.553 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:29:31.553 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:29:31.553 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:29:31.553 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:29:31.553 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:29:31.553 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:29:31.553 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:29:31.553 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:29:31.553 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:29:31.553 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:29:31.553 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:29:31.554 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:29:31.554 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:29:31.554 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:29:31.554 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:29:31.554 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:29:31.554 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:29:31.554 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:29:31.554 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:29:31.554 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:29:31.554 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:29:31.554 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:29:31.554 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:29:31.554 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:29:31.554 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:29:31.555 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:29:31.555 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:29:31.555 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:29:31.555 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:29:31.555 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:29:31.555 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:29:32.660 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:29:32.660 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:29:32.660 INFO:teuthology.orchestra.run.smithi122.stdout:Upgraded: 2024-03-25T16:29:32.660 INFO:teuthology.orchestra.run.smithi122.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:32.660 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:32.660 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:29:32.661 INFO:teuthology.orchestra.run.smithi122.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:29:32.661 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:32.661 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:32.661 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:32.661 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:32.661 INFO:teuthology.orchestra.run.smithi122.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:29:32.661 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:29:32.661 INFO:teuthology.orchestra.run.smithi122.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:29:32.661 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:32.661 INFO:teuthology.orchestra.run.smithi122.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:29:32.661 INFO:teuthology.orchestra.run.smithi122.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:29:32.661 INFO:teuthology.orchestra.run.smithi122.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:29:32.661 INFO:teuthology.orchestra.run.smithi122.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:32.661 INFO:teuthology.orchestra.run.smithi122.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:29:32.662 INFO:teuthology.orchestra.run.smithi122.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:32.662 INFO:teuthology.orchestra.run.smithi122.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:29:32.662 INFO:teuthology.orchestra.run.smithi122.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:29:32.662 INFO:teuthology.orchestra.run.smithi122.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:29:32.662 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:32.662 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:32.662 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:32.662 INFO:teuthology.orchestra.run.smithi122.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:29:32.662 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:32.662 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:32.662 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:32.662 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:29:32.662 INFO:teuthology.orchestra.run.smithi122.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:29:32.662 INFO:teuthology.orchestra.run.smithi122.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:29:32.663 INFO:teuthology.orchestra.run.smithi122.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:29:32.663 INFO:teuthology.orchestra.run.smithi122.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:29:32.663 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:29:32.663 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:29:33.019 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-test 2024-03-25T16:29:33.582 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:01:15 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:29:33.729 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:29:33.729 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:29:33.729 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-25T16:29:33.729 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:29:33.729 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:29:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:29:33.730 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-25T16:29:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:29:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:29:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:29:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:29:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:29:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:29:33.730 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:29:33.730 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:29:33.730 INFO:teuthology.orchestra.run.smithi122.stdout:Install 6 Packages 2024-03-25T16:29:33.730 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:29:33.731 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 48 M 2024-03-25T16:29:33.731 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 204 M 2024-03-25T16:29:33.731 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:29:33.939 INFO:teuthology.orchestra.run.smithi122.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T16:29:33.990 INFO:teuthology.orchestra.run.smithi122.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T16:29:34.023 INFO:teuthology.orchestra.run.smithi122.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T16:29:34.699 INFO:teuthology.orchestra.run.smithi122.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 177 kB/s | 163 kB 00:00 2024-03-25T16:29:35.099 INFO:teuthology.orchestra.run.smithi122.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 174 kB/s | 187 kB 00:01 2024-03-25T16:29:36.672 INFO:teuthology.orchestra.run.smithi204.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 285 kB/s | 20 MB 01:11 2024-03-25T16:29:36.676 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:36.676 INFO:teuthology.orchestra.run.smithi204.stdout:Total 826 kB/s | 58 MB 01:11 2024-03-25T16:29:36.708 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:29:36.745 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:29:36.745 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:29:37.226 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:29:37.226 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:29:38.739 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:29:38.822 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:29:38.852 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:29:38.885 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:29:38.908 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:29:38.925 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:29:39.098 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:29:39.156 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:29:39.497 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:29:39.563 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:29:39.655 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:29:39.744 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:29:39.782 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:29:39.827 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:29:39.856 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:29:39.900 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:29:39.969 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:29:40.000 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:29:40.062 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:29:40.090 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:29:40.201 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:29:40.233 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:29:40.273 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:29:40.306 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:29:40.335 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:29:40.359 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:29:40.668 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:29:40.698 INFO:teuthology.orchestra.run.smithi204.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:29:40.787 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:29:40.819 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:29:40.857 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:29:41.310 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:29:41.342 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:29:41.413 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:29:41.481 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:29:41.511 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:29:43.726 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:29:43.766 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:29:44.317 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:29:44.361 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:29:44.455 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:29:44.455 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:29:44.455 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:29:44.456 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:29:44.496 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:29:53.858 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:29:53.858 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /sys 2024-03-25T16:29:53.858 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /proc 2024-03-25T16:29:53.858 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /mnt 2024-03-25T16:29:53.858 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /var/tmp 2024-03-25T16:29:53.858 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /home 2024-03-25T16:29:53.858 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /root 2024-03-25T16:29:53.859 INFO:teuthology.orchestra.run.smithi204.stdout:skipping the directory /tmp 2024-03-25T16:29:53.859 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:29:54.771 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:29:54.796 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:29:54.796 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:29:54.796 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:29:54.796 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:29:54.797 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:29:54.797 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:29:54.798 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:29:54.866 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:29:54.866 INFO:teuthology.orchestra.run.smithi204.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:29:55.991 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:29:55.991 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:29:55.991 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:29:55.991 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:29:55.991 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:29:55.991 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:29:55.991 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:29:55.991 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:29:55.991 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:29:55.992 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:29:55.992 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:29:55.992 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:29:55.992 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:29:55.992 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:29:55.992 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:29:55.992 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:29:55.992 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:29:55.992 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:29:55.992 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:29:55.992 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:29:55.992 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:29:55.992 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:29:55.992 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:29:55.993 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:29:55.993 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:29:55.993 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:29:55.993 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:29:55.993 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:29:55.993 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:29:55.993 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:29:55.993 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:29:55.994 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:29:55.994 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:29:55.994 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:29:57.119 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:29:57.120 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:29:57.120 INFO:teuthology.orchestra.run.smithi204.stdout:Upgraded: 2024-03-25T16:29:57.120 INFO:teuthology.orchestra.run.smithi204.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.120 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.120 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:29:57.120 INFO:teuthology.orchestra.run.smithi204.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:29:57.120 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.120 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.120 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.120 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.120 INFO:teuthology.orchestra.run.smithi204.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:29:57.120 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:29:57.120 INFO:teuthology.orchestra.run.smithi204.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:29:57.120 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.120 INFO:teuthology.orchestra.run.smithi204.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:29:57.121 INFO:teuthology.orchestra.run.smithi204.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:29:57.121 INFO:teuthology.orchestra.run.smithi204.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:29:57.121 INFO:teuthology.orchestra.run.smithi204.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.121 INFO:teuthology.orchestra.run.smithi204.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:29:57.121 INFO:teuthology.orchestra.run.smithi204.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.121 INFO:teuthology.orchestra.run.smithi204.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:29:57.121 INFO:teuthology.orchestra.run.smithi204.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:29:57.121 INFO:teuthology.orchestra.run.smithi204.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:29:57.121 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.121 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.121 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.121 INFO:teuthology.orchestra.run.smithi204.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:29:57.121 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.121 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.121 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:57.122 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:29:57.122 INFO:teuthology.orchestra.run.smithi204.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:29:57.122 INFO:teuthology.orchestra.run.smithi204.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:29:57.122 INFO:teuthology.orchestra.run.smithi204.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:29:57.122 INFO:teuthology.orchestra.run.smithi204.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:29:57.122 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:29:57.122 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:29:57.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-test 2024-03-25T16:29:58.033 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:01:36 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:29:58.184 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:29:58.184 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:29:58.184 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-25T16:29:58.184 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:29:58.184 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:29:58.184 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:29:58.184 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-25T16:29:58.184 INFO:teuthology.orchestra.run.smithi204.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:29:58.184 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:29:58.184 INFO:teuthology.orchestra.run.smithi204.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:29:58.185 INFO:teuthology.orchestra.run.smithi204.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:29:58.185 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:29:58.185 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:29:58.185 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:29:58.185 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:29:58.185 INFO:teuthology.orchestra.run.smithi204.stdout:Install 6 Packages 2024-03-25T16:29:58.185 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:29:58.185 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 48 M 2024-03-25T16:29:58.185 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 204 M 2024-03-25T16:29:58.185 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:29:58.394 INFO:teuthology.orchestra.run.smithi204.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T16:29:58.444 INFO:teuthology.orchestra.run.smithi204.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T16:29:58.478 INFO:teuthology.orchestra.run.smithi204.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-25T16:29:58.562 INFO:teuthology.orchestra.run.smithi204.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.2 MB/s | 187 kB 00:00 2024-03-25T16:29:59.146 INFO:teuthology.orchestra.run.smithi204.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 178 kB/s | 163 kB 00:00 2024-03-25T16:30:45.306 INFO:teuthology.orchestra.run.smithi122.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 677 kB/s | 47 MB 01:11 2024-03-25T16:30:45.307 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:30:45.308 INFO:teuthology.orchestra.run.smithi122.stdout:Total 689 kB/s | 48 MB 01:11 2024-03-25T16:30:45.319 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:30:45.338 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:30:45.339 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:30:45.764 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:30:45.764 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:30:46.342 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:30:46.422 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:30:46.465 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:30:46.505 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:30:46.558 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:30:46.583 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:30:51.080 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:30:51.167 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:30:53.595 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:30:53.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:30:53.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:30:53.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:30:53.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:30:53.596 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:30:53.933 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:30:53.934 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:30:53.934 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:30:53.934 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:53.934 INFO:teuthology.orchestra.run.smithi122.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:30:53.934 INFO:teuthology.orchestra.run.smithi122.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:30:53.934 INFO:teuthology.orchestra.run.smithi122.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:30:53.934 INFO:teuthology.orchestra.run.smithi122.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:30:53.934 INFO:teuthology.orchestra.run.smithi122.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:30:53.934 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:30:53.934 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:30:54.262 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph 2024-03-25T16:30:54.825 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:02:36 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:30:54.976 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:30:54.979 INFO:teuthology.orchestra.run.smithi122.stdout:========================================================================================= 2024-03-25T16:30:54.979 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-25T16:30:54.979 INFO:teuthology.orchestra.run.smithi122.stdout:========================================================================================= 2024-03-25T16:30:54.979 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:30:54.979 INFO:teuthology.orchestra.run.smithi122.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:30:54.979 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-25T16:30:54.979 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:30:54.980 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:30:54.980 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:30:54.980 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:30:54.980 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:30:54.980 INFO:teuthology.orchestra.run.smithi122.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:30:54.980 INFO:teuthology.orchestra.run.smithi122.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:30:54.980 INFO:teuthology.orchestra.run.smithi122.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:30:54.980 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:30:54.980 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:30:54.980 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:30:54.980 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:30:54.980 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:30:54.980 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:30:54.981 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:30:54.981 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:30:54.981 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:30:54.981 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:30:54.981 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:30:54.981 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:30:54.981 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:30:54.981 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:30:54.981 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:30:54.981 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:30:54.981 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:30:54.981 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:30:54.981 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:30:54.982 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:30:54.982 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:30:54.982 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:30:54.982 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:30:54.982 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:30:54.982 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:30:54.982 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:30:54.982 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:30:54.982 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:30:54.982 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:30:54.982 INFO:teuthology.orchestra.run.smithi122.stdout:========================================================================================= 2024-03-25T16:30:54.982 INFO:teuthology.orchestra.run.smithi122.stdout:Install 36 Packages 2024-03-25T16:30:54.982 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:30:54.983 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 30 M 2024-03-25T16:30:54.983 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 107 M 2024-03-25T16:30:54.983 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:30:56.316 INFO:teuthology.orchestra.run.smithi122.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 9.8 kB/s | 6.6 kB 00:00 2024-03-25T16:30:57.599 INFO:teuthology.orchestra.run.smithi122.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 780 kB/s | 1.5 MB 00:01 2024-03-25T16:30:58.975 INFO:teuthology.orchestra.run.smithi122.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 653 kB/s | 2.1 MB 00:03 2024-03-25T16:30:59.375 INFO:teuthology.orchestra.run.smithi122.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 618 kB/s | 247 kB 00:00 2024-03-25T16:30:59.492 INFO:teuthology.orchestra.run.smithi122.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T16:30:59.534 INFO:teuthology.orchestra.run.smithi122.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:30:59.567 INFO:teuthology.orchestra.run.smithi122.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 955 kB/s | 32 kB 00:00 2024-03-25T16:30:59.601 INFO:teuthology.orchestra.run.smithi122.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:30:59.751 INFO:teuthology.orchestra.run.smithi122.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 257 kB/s | 39 kB 00:00 2024-03-25T16:30:59.802 INFO:teuthology.orchestra.run.smithi122.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.6 MB/s | 241 kB 00:00 2024-03-25T16:30:59.877 INFO:teuthology.orchestra.run.smithi122.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T16:30:59.910 INFO:teuthology.orchestra.run.smithi122.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:30:59.953 INFO:teuthology.orchestra.run.smithi122.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T16:30:59.995 INFO:teuthology.orchestra.run.smithi122.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:31:00.203 INFO:teuthology.orchestra.run.smithi122.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 599 kB/s | 125 kB 00:00 2024-03-25T16:31:00.778 INFO:teuthology.orchestra.run.smithi122.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 39 kB/s | 22 kB 00:00 2024-03-25T16:31:01.170 INFO:teuthology.orchestra.run.smithi122.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 386 kB/s | 151 kB 00:00 2024-03-25T16:31:02.279 INFO:teuthology.orchestra.run.smithi122.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 39 kB/s | 43 kB 00:01 2024-03-25T16:31:03.920 INFO:teuthology.orchestra.run.smithi122.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 105 kB/s | 172 kB 00:01 2024-03-25T16:31:04.579 INFO:teuthology.orchestra.run.smithi122.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 544 kB/s | 358 kB 00:00 2024-03-25T16:31:04.938 INFO:teuthology.orchestra.run.smithi122.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 30 kB/s | 11 kB 00:00 2024-03-25T16:31:06.529 INFO:teuthology.orchestra.run.smithi122.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 11 kB/s | 18 kB 00:01 2024-03-25T16:31:06.771 INFO:teuthology.orchestra.run.smithi122.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 96 kB/s | 23 kB 00:00 2024-03-25T16:31:06.871 INFO:teuthology.orchestra.run.smithi122.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 195 kB/s | 19 kB 00:00 2024-03-25T16:31:06.913 INFO:teuthology.orchestra.run.smithi122.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 471 kB/s | 20 kB 00:00 2024-03-25T16:31:06.972 INFO:teuthology.orchestra.run.smithi122.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 794 kB/s | 46 kB 00:00 2024-03-25T16:31:07.022 INFO:teuthology.orchestra.run.smithi122.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-25T16:31:07.098 INFO:teuthology.orchestra.run.smithi122.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.5 MB/s | 272 kB 00:00 2024-03-25T16:31:07.281 INFO:teuthology.orchestra.run.smithi122.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 89 kB/s | 16 kB 00:00 2024-03-25T16:31:07.456 INFO:teuthology.orchestra.run.smithi122.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 514 kB/s | 90 kB 00:00 2024-03-25T16:31:07.682 INFO:teuthology.orchestra.run.smithi122.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 159 kB/s | 36 kB 00:00 2024-03-25T16:31:07.765 INFO:teuthology.orchestra.run.smithi122.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-03-25T16:31:07.874 INFO:teuthology.orchestra.run.smithi122.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-03-25T16:31:07.924 INFO:teuthology.orchestra.run.smithi122.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 400 kB/s | 20 kB 00:00 2024-03-25T16:31:18.932 INFO:teuthology.orchestra.run.smithi122.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 212 kB/s | 4.7 MB 00:22 2024-03-25T16:31:31.439 INFO:teuthology.orchestra.run.smithi122.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 508 kB/s | 17 MB 00:33 2024-03-25T16:31:31.443 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:31:31.443 INFO:teuthology.orchestra.run.smithi122.stdout:Total 833 kB/s | 30 MB 00:36 2024-03-25T16:31:31.465 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:31:31.508 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:31:31.508 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:31:31.942 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:31:31.942 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:31:33.281 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:31:33.333 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:31:33.372 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:31:33.411 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:31:33.482 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:31:33.526 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:31:33.559 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:31:33.651 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:31:33.779 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:31:33.820 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:31:33.901 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:31:33.955 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:31:33.993 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:31:34.042 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:31:34.097 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:31:34.284 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:31:34.330 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:31:34.361 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:31:34.395 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:31:34.425 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:31:34.501 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:31:34.544 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:31:34.577 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:31:34.618 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:31:34.652 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:31:34.695 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:31:34.785 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:31:34.826 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:31:34.898 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:31:35.037 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:31:35.161 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:31:35.318 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:31:35.348 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:31:35.371 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:31:35.371 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:31:35.371 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:31:35.372 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:31:35.372 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:31:35.372 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:36.799 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:31:36.825 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:31:36.825 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:31:36.825 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:31:36.825 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:31:36.825 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:31:36.825 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:37.266 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:31:37.289 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:31:37.289 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:31:37.289 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:31:37.289 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:31:37.289 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:31:37.289 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:37.492 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:31:37.514 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:31:37.514 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:31:37.514 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:31:37.514 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:31:37.514 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:31:37.515 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:37.580 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:31:39.215 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:31:39.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:31:39.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:31:39.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:31:39.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:31:39.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:31:39.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:31:39.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:31:39.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:31:39.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:31:39.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:31:39.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:31:39.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:31:39.216 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:31:39.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:31:39.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:31:39.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:31:39.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:31:39.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:31:39.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:31:39.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:31:39.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:31:39.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:31:39.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:31:39.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:31:39.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:31:39.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:31:39.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:31:39.217 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:31:39.218 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:31:39.218 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:31:39.218 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:31:39.218 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:31:39.218 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:31:39.218 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:31:39.218 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:31:40.232 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:31:40.232 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:40.232 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:31:40.232 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:40.232 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:40.232 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:40.232 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:31:40.232 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:40.233 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:40.233 INFO:teuthology.orchestra.run.smithi122.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:31:40.233 INFO:teuthology.orchestra.run.smithi122.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:31:40.233 INFO:teuthology.orchestra.run.smithi122.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:31:40.233 INFO:teuthology.orchestra.run.smithi122.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:31:40.233 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:31:40.233 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:31:40.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:31:40.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:31:40.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:31:40.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:31:40.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:31:40.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:31:40.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:31:40.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:31:40.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:31:40.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:31:40.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:31:40.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:31:40.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:31:40.234 INFO:teuthology.orchestra.run.smithi122.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:31:40.235 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:31:40.235 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:31:40.235 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:31:40.235 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:31:40.235 INFO:teuthology.orchestra.run.smithi122.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:31:40.235 INFO:teuthology.orchestra.run.smithi122.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:31:40.235 INFO:teuthology.orchestra.run.smithi122.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:31:40.235 INFO:teuthology.orchestra.run.smithi122.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:31:40.235 INFO:teuthology.orchestra.run.smithi122.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:31:40.235 INFO:teuthology.orchestra.run.smithi122.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:31:40.235 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:40.235 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:31:40.588 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-base 2024-03-25T16:31:41.151 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:03:23 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:31:41.248 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:31:41.300 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:31:41.301 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-25T16:31:41.301 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:31:41.380 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install cephadm 2024-03-25T16:31:41.937 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:03:23 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:31:42.085 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:31:42.086 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:31:42.086 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-25T16:31:42.086 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:31:42.087 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:31:42.087 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:31:42.087 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:42.087 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:31:42.087 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:31:42.087 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T16:31:42.087 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:42.087 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 747 k 2024-03-25T16:31:42.087 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 754 k 2024-03-25T16:31:42.087 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:31:45.117 INFO:teuthology.orchestra.run.smithi122.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 247 kB/s | 747 kB 00:03 2024-03-25T16:31:45.117 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:31:45.118 INFO:teuthology.orchestra.run.smithi122.stdout:Total 247 kB/s | 747 kB 00:03 2024-03-25T16:31:45.118 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:31:45.124 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:31:45.124 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:31:45.130 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:31:45.130 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:31:45.210 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:31:45.637 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:31:45.729 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:31:46.192 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:31:46.478 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:31:46.478 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:46.478 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:31:46.478 INFO:teuthology.orchestra.run.smithi122.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:31:46.478 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:46.478 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:31:46.759 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:31:47.327 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:03:29 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:31:47.481 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:31:47.482 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:31:47.482 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T16:31:47.482 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:31:47.482 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:31:47.482 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:31:47.482 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:47.482 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:31:47.482 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:31:47.482 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T16:31:47.482 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:47.482 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 145 k 2024-03-25T16:31:47.483 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 436 k 2024-03-25T16:31:47.483 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:31:48.423 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 154 kB/s | 145 kB 00:00 2024-03-25T16:31:48.424 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:31:48.424 INFO:teuthology.orchestra.run.smithi122.stdout:Total 154 kB/s | 145 kB 00:00 2024-03-25T16:31:48.425 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:31:48.432 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:31:48.433 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:31:48.486 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:31:48.486 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:31:48.622 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:31:48.682 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:31:48.707 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:31:48.707 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:31:48.707 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:31:48.707 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:49.678 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:31:49.678 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:49.678 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:31:49.678 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:49.678 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:49.678 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:31:49.931 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr 2024-03-25T16:31:50.505 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:03:32 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:31:50.603 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:31:50.655 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:31:50.656 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-25T16:31:50.656 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:31:50.733 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:31:51.290 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:03:33 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:31:51.438 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:31:51.439 INFO:teuthology.orchestra.run.smithi122.stdout:====================================================================================== 2024-03-25T16:31:51.440 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-25T16:31:51.440 INFO:teuthology.orchestra.run.smithi122.stdout:====================================================================================== 2024-03-25T16:31:51.440 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:31:51.440 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:31:51.440 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-25T16:31:51.440 INFO:teuthology.orchestra.run.smithi122.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:31:51.440 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:31:51.440 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:31:51.440 INFO:teuthology.orchestra.run.smithi122.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:31:51.440 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:31:51.440 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:31:51.440 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:31:51.440 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:31:51.441 INFO:teuthology.orchestra.run.smithi122.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:31:51.441 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:31:51.441 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:31:51.441 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:51.441 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:31:51.441 INFO:teuthology.orchestra.run.smithi122.stdout:====================================================================================== 2024-03-25T16:31:51.441 INFO:teuthology.orchestra.run.smithi122.stdout:Install 12 Packages 2024-03-25T16:31:51.441 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:51.442 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 6.9 M 2024-03-25T16:31:51.442 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 26 M 2024-03-25T16:31:51.442 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:31:52.428 INFO:teuthology.orchestra.run.smithi122.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 27 kB/s | 15 kB 00:00 2024-03-25T16:31:52.453 INFO:teuthology.orchestra.run.smithi122.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 53 kB/s | 31 kB 00:00 2024-03-25T16:31:52.611 INFO:teuthology.orchestra.run.smithi122.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-25T16:31:52.645 INFO:teuthology.orchestra.run.smithi122.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-25T16:31:53.045 INFO:teuthology.orchestra.run.smithi122.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.3 MB/s | 548 kB 00:00 2024-03-25T16:31:53.112 INFO:teuthology.orchestra.run.smithi122.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-03-25T16:31:53.196 INFO:teuthology.orchestra.run.smithi122.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.4 MB/s | 863 kB 00:00 2024-03-25T16:31:53.304 INFO:teuthology.orchestra.run.smithi122.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 11 MB/s | 2.0 MB 00:00 2024-03-25T16:31:53.375 INFO:teuthology.orchestra.run.smithi122.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 466 kB/s | 31 kB 00:00 2024-03-25T16:31:53.438 INFO:teuthology.orchestra.run.smithi122.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-25T16:31:53.508 INFO:teuthology.orchestra.run.smithi122.stdout:(11/12): python3-grpcio-tools-1.46.7-10.el9.x86 467 kB/s | 144 kB 00:00 2024-03-25T16:31:56.497 INFO:teuthology.orchestra.run.smithi122.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 393 kB/s | 1.8 MB 00:04 2024-03-25T16:31:56.503 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:31:56.503 INFO:teuthology.orchestra.run.smithi122.stdout:Total 1.4 MB/s | 6.9 MB 00:05 2024-03-25T16:31:56.523 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:31:56.542 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:31:56.542 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:31:56.729 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:31:56.729 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:31:57.254 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:31:57.339 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:31:57.382 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:31:57.428 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:31:57.468 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:31:57.572 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:31:57.700 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:31:57.794 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:31:57.839 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:31:57.868 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:31:57.899 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:31:58.139 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:31:58.175 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:31:58.950 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:31:58.950 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:31:58.950 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:31:58.950 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:31:58.950 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:31:58.950 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:31:58.950 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:31:58.950 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:31:58.950 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:31:58.950 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:31:58.951 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:31:58.951 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:31:59.310 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:31:59.310 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:59.310 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:31:59.310 INFO:teuthology.orchestra.run.smithi122.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:31:59.310 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:31:59.310 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:31:59.310 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:31:59.310 INFO:teuthology.orchestra.run.smithi122.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:31:59.310 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:31:59.310 INFO:teuthology.orchestra.run.smithi122.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:31:59.310 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:31:59.311 INFO:teuthology.orchestra.run.smithi122.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:31:59.311 INFO:teuthology.orchestra.run.smithi122.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:31:59.311 INFO:teuthology.orchestra.run.smithi122.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:31:59.311 INFO:teuthology.orchestra.run.smithi122.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:31:59.311 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:31:59.311 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:31:59.585 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:32:00.143 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:03:42 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:32:00.292 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:32:00.293 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================================ 2024-03-25T16:32:00.293 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-25T16:32:00.293 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================================ 2024-03-25T16:32:00.293 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:32:00.293 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:32:00.294 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-25T16:32:00.294 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:32:00.294 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:32:00.294 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:32:00.294 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:32:00.294 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:32:00.294 INFO:teuthology.orchestra.run.smithi122.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:32:00.294 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:32:00.294 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:32:00.294 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:32:00.294 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:32:00.294 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:32:00.294 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:00.294 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:32:00.295 INFO:teuthology.orchestra.run.smithi122.stdout:============================================================================================ 2024-03-25T16:32:00.295 INFO:teuthology.orchestra.run.smithi122.stdout:Install 12 Packages 2024-03-25T16:32:00.295 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:00.295 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 37 M 2024-03-25T16:32:00.295 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 209 M 2024-03-25T16:32:00.295 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:32:00.504 INFO:teuthology.orchestra.run.smithi122.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 248 kB/s | 32 kB 00:00 2024-03-25T16:32:00.562 INFO:teuthology.orchestra.run.smithi122.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 490 kB/s | 17 kB 00:00 2024-03-25T16:32:00.604 INFO:teuthology.orchestra.run.smithi122.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 822 kB/s | 34 kB 00:00 2024-03-25T16:32:00.780 INFO:teuthology.orchestra.run.smithi122.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-03-25T16:32:00.880 INFO:teuthology.orchestra.run.smithi122.stdout:(5/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.9 MB/s | 3.0 MB 00:00 2024-03-25T16:32:00.947 INFO:teuthology.orchestra.run.smithi122.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-25T16:32:00.980 INFO:teuthology.orchestra.run.smithi122.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.2 MB/s | 217 kB 00:00 2024-03-25T16:32:01.257 INFO:teuthology.orchestra.run.smithi204.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 393 kB/s | 47 MB 02:03 2024-03-25T16:32:01.259 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:32:01.260 INFO:teuthology.orchestra.run.smithi204.stdout:Total 401 kB/s | 48 MB 02:03 2024-03-25T16:32:01.271 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:32:01.292 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:32:01.292 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:32:01.356 INFO:teuthology.orchestra.run.smithi122.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-25T16:32:01.670 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:32:01.670 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:32:01.672 INFO:teuthology.orchestra.run.smithi122.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 617 kB/s | 195 kB 00:00 2024-03-25T16:32:01.748 INFO:teuthology.orchestra.run.smithi122.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-03-25T16:32:01.773 INFO:teuthology.orchestra.run.smithi122.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 803 kB 00:00 2024-03-25T16:32:02.262 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:32:02.324 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:32:02.365 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:32:02.406 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:32:02.461 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:32:02.486 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:32:07.018 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:32:07.102 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:32:09.420 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:32:09.420 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:32:09.420 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:32:09.421 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:32:09.421 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:32:09.421 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:32:09.785 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:32:09.785 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:32:09.785 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:32:09.785 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:09.786 INFO:teuthology.orchestra.run.smithi204.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:32:09.786 INFO:teuthology.orchestra.run.smithi204.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:09.786 INFO:teuthology.orchestra.run.smithi204.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:32:09.786 INFO:teuthology.orchestra.run.smithi204.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:32:09.786 INFO:teuthology.orchestra.run.smithi204.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:32:09.786 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:32:09.786 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:32:10.093 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph 2024-03-25T16:32:10.663 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:03:48 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:32:10.821 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:32:10.824 INFO:teuthology.orchestra.run.smithi204.stdout:========================================================================================= 2024-03-25T16:32:10.824 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-25T16:32:10.824 INFO:teuthology.orchestra.run.smithi204.stdout:========================================================================================= 2024-03-25T16:32:10.824 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:32:10.824 INFO:teuthology.orchestra.run.smithi204.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:32:10.824 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-25T16:32:10.824 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:32:10.824 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:32:10.824 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:32:10.824 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:32:10.825 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:32:10.825 INFO:teuthology.orchestra.run.smithi204.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:32:10.825 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:32:10.825 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:32:10.825 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:32:10.825 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:32:10.825 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:32:10.825 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:32:10.825 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:32:10.825 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:32:10.825 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:32:10.825 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:32:10.825 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:32:10.825 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:32:10.825 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:32:10.826 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:32:10.826 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:32:10.826 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:32:10.826 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:32:10.826 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:32:10.826 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:32:10.826 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:32:10.826 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:32:10.826 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:32:10.826 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:32:10.826 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:32:10.826 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:32:10.826 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:32:10.827 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:32:10.827 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:32:10.827 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:32:10.827 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:32:10.827 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:32:10.827 INFO:teuthology.orchestra.run.smithi204.stdout:========================================================================================= 2024-03-25T16:32:10.827 INFO:teuthology.orchestra.run.smithi204.stdout:Install 36 Packages 2024-03-25T16:32:10.827 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:32:10.828 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 30 M 2024-03-25T16:32:10.828 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 107 M 2024-03-25T16:32:10.828 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:32:11.637 INFO:teuthology.orchestra.run.smithi204.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T16:32:15.554 INFO:teuthology.orchestra.run.smithi204.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 500 kB/s | 2.1 MB 00:04 2024-03-25T16:32:16.045 INFO:teuthology.orchestra.run.smithi204.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 315 kB/s | 1.5 MB 00:04 2024-03-25T16:32:16.912 INFO:teuthology.orchestra.run.smithi204.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 285 kB/s | 247 kB 00:00 2024-03-25T16:32:17.037 INFO:teuthology.orchestra.run.smithi204.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T16:32:17.081 INFO:teuthology.orchestra.run.smithi204.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:32:17.105 INFO:teuthology.orchestra.run.smithi204.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T16:32:17.130 INFO:teuthology.orchestra.run.smithi204.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-25T16:32:17.155 INFO:teuthology.orchestra.run.smithi204.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T16:32:17.197 INFO:teuthology.orchestra.run.smithi204.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T16:32:17.264 INFO:teuthology.orchestra.run.smithi204.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T16:32:17.298 INFO:teuthology.orchestra.run.smithi204.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:32:17.340 INFO:teuthology.orchestra.run.smithi204.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 2.9 MB/s | 124 kB 00:00 2024-03-25T16:32:17.373 INFO:teuthology.orchestra.run.smithi204.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T16:32:17.507 INFO:teuthology.orchestra.run.smithi204.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 937 kB/s | 125 kB 00:00 2024-03-25T16:32:17.766 INFO:teuthology.orchestra.run.smithi204.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2024-03-25T16:32:18.241 INFO:teuthology.orchestra.run.smithi204.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 318 kB/s | 151 kB 00:00 2024-03-25T16:32:18.366 INFO:teuthology.orchestra.run.smithi204.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 348 kB/s | 43 kB 00:00 2024-03-25T16:32:18.516 INFO:teuthology.orchestra.run.smithi204.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-03-25T16:32:18.709 INFO:teuthology.orchestra.run.smithi204.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-03-25T16:32:18.809 INFO:teuthology.orchestra.run.smithi204.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-25T16:32:18.909 INFO:teuthology.orchestra.run.smithi204.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 177 kB/s | 18 kB 00:00 2024-03-25T16:32:19.009 INFO:teuthology.orchestra.run.smithi204.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-25T16:32:19.110 INFO:teuthology.orchestra.run.smithi204.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 194 kB/s | 19 kB 00:00 2024-03-25T16:32:19.210 INFO:teuthology.orchestra.run.smithi204.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 197 kB/s | 20 kB 00:00 2024-03-25T16:32:19.318 INFO:teuthology.orchestra.run.smithi204.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 427 kB/s | 46 kB 00:00 2024-03-25T16:32:19.427 INFO:teuthology.orchestra.run.smithi204.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 726 kB/s | 79 kB 00:00 2024-03-25T16:32:19.553 INFO:teuthology.orchestra.run.smithi204.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-25T16:32:19.653 INFO:teuthology.orchestra.run.smithi204.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 164 kB/s | 16 kB 00:00 2024-03-25T16:32:19.762 INFO:teuthology.orchestra.run.smithi204.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 831 kB/s | 90 kB 00:00 2024-03-25T16:32:19.862 INFO:teuthology.orchestra.run.smithi204.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 357 kB/s | 36 kB 00:00 2024-03-25T16:32:19.979 INFO:teuthology.orchestra.run.smithi204.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-03-25T16:32:20.121 INFO:teuthology.orchestra.run.smithi204.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-03-25T16:32:20.221 INFO:teuthology.orchestra.run.smithi204.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-25T16:32:26.254 INFO:teuthology.orchestra.run.smithi122.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 292 kB/s | 7.4 MB 00:25 2024-03-25T16:32:26.256 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:32:26.257 INFO:teuthology.orchestra.run.smithi122.stdout:Total 1.4 MB/s | 37 MB 00:25 2024-03-25T16:32:26.299 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:32:26.324 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:32:26.324 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:32:26.681 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:32:26.681 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:32:27.382 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:32:27.452 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:32:27.491 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:32:27.584 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:32:27.670 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:32:27.848 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:32:27.877 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:32:27.954 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:32:28.607 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:32:28.723 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:32:29.164 INFO:teuthology.orchestra.run.smithi204.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 274 kB/s | 4.7 MB 00:17 2024-03-25T16:32:29.964 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:32:30.979 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:32:31.002 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:32:33.592 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:32:33.593 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:32:33.593 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:32:33.593 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:32:34.078 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:32:34.078 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:34.078 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:32:34.078 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:32:34.078 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:32:34.078 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:32:34.078 INFO:teuthology.orchestra.run.smithi122.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:32:34.078 INFO:teuthology.orchestra.run.smithi122.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:32:34.078 INFO:teuthology.orchestra.run.smithi122.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:32:34.079 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:32:34.079 INFO:teuthology.orchestra.run.smithi122.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:32:34.079 INFO:teuthology.orchestra.run.smithi122.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:32:34.079 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:32:34.079 INFO:teuthology.orchestra.run.smithi122.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:32:34.079 INFO:teuthology.orchestra.run.smithi122.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:32:34.079 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:34.079 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:32:34.489 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:32:35.056 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:04:17 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:32:35.205 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:32:35.207 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2024-03-25T16:32:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-25T16:32:35.207 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2024-03-25T16:32:35.207 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:32:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:32:35.207 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-25T16:32:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:32:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:32:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:32:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:32:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:32:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:32:35.207 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:32:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:32:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:32:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:32:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:32:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:32:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:35.208 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:32:35.208 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2024-03-25T16:32:35.208 INFO:teuthology.orchestra.run.smithi122.stdout:Install 13 Packages 2024-03-25T16:32:35.208 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:35.209 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 2.0 M 2024-03-25T16:32:35.209 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 26 M 2024-03-25T16:32:35.209 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:32:35.620 INFO:teuthology.orchestra.run.smithi122.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 122 kB/s | 24 kB 00:00 2024-03-25T16:32:35.645 INFO:teuthology.orchestra.run.smithi122.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 81 kB/s | 18 kB 00:00 2024-03-25T16:32:35.779 INFO:teuthology.orchestra.run.smithi122.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.1 MB/s | 174 kB 00:00 2024-03-25T16:32:35.804 INFO:teuthology.orchestra.run.smithi122.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 833 kB/s | 132 kB 00:00 2024-03-25T16:32:35.830 INFO:teuthology.orchestra.run.smithi122.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T16:32:35.847 INFO:teuthology.orchestra.run.smithi122.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-25T16:32:35.947 INFO:teuthology.orchestra.run.smithi122.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 99 kB/s | 52 kB 00:00 2024-03-25T16:32:36.322 INFO:teuthology.orchestra.run.smithi122.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 65 kB/s | 32 kB 00:00 2024-03-25T16:32:36.364 INFO:teuthology.orchestra.run.smithi122.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 29 kB/s | 15 kB 00:00 2024-03-25T16:32:36.556 INFO:teuthology.orchestra.run.smithi122.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 308 kB/s | 59 kB 00:00 2024-03-25T16:32:36.723 INFO:teuthology.orchestra.run.smithi122.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 278 kB/s | 203 kB 00:00 2024-03-25T16:32:36.773 INFO:teuthology.orchestra.run.smithi122.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 413 kB/s | 90 kB 00:00 2024-03-25T16:32:37.932 INFO:teuthology.orchestra.run.smithi122.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 650 kB/s | 1.0 MB 00:01 2024-03-25T16:32:37.936 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:32:37.937 INFO:teuthology.orchestra.run.smithi122.stdout:Total 769 kB/s | 2.0 MB 00:02 2024-03-25T16:32:37.937 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:32:37.952 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:32:37.952 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:32:38.048 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:32:38.049 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:32:38.565 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:32:38.743 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:32:38.799 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:32:38.854 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:32:38.883 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:32:38.925 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:32:39.017 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:32:39.135 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:32:39.185 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:32:39.896 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:32:39.948 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:32:39.999 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:32:40.047 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:32:40.103 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:32:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:32:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:32:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:32:40.856 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:32:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:32:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:32:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:32:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:32:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:32:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:32:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:32:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:32:40.857 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:32:41.220 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:32:41.221 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:41.221 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:32:41.221 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:32:41.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:32:41.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:32:41.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:32:41.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:32:41.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:32:41.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:32:41.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:32:41.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:32:41.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:32:41.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:32:41.221 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:32:41.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:32:41.222 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:41.222 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:32:41.486 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:32:42.051 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:04:24 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:32:42.203 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:32:42.203 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2024-03-25T16:32:42.203 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-25T16:32:42.204 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2024-03-25T16:32:42.204 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:32:42.204 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:32:42.204 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-25T16:32:42.204 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:32:42.204 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:32:42.204 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:32:42.204 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:32:42.204 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:32:42.204 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:42.204 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:32:42.204 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2024-03-25T16:32:42.204 INFO:teuthology.orchestra.run.smithi122.stdout:Install 6 Packages 2024-03-25T16:32:42.204 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:42.205 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 6.8 M 2024-03-25T16:32:42.205 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 33 M 2024-03-25T16:32:42.205 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:32:42.875 INFO:teuthology.orchestra.run.smithi122.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-03-25T16:32:43.059 INFO:teuthology.orchestra.run.smithi122.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 5.8 MB 00:00 2024-03-25T16:32:43.451 INFO:teuthology.orchestra.run.smithi122.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 190 kB/s | 138 kB 00:00 2024-03-25T16:32:43.496 INFO:teuthology.orchestra.run.smithi122.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 132 kB/s | 58 kB 00:00 2024-03-25T16:32:43.576 INFO:teuthology.orchestra.run.smithi122.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 782 kB/s | 548 kB 00:00 2024-03-25T16:32:43.868 INFO:teuthology.orchestra.run.smithi122.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 122 kB/s | 51 kB 00:00 2024-03-25T16:32:43.871 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:32:43.871 INFO:teuthology.orchestra.run.smithi122.stdout:Total 4.1 MB/s | 6.8 MB 00:01 2024-03-25T16:32:43.886 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:32:43.894 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:32:43.894 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:32:43.983 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:32:43.983 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:32:44.275 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:32:44.427 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:32:44.470 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:32:44.982 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:32:45.064 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:32:45.126 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:32:45.149 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:32:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:32:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:32:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:32:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:32:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:32:45.931 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:32:46.222 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:32:46.222 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:46.222 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:32:46.222 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:32:46.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:32:46.222 INFO:teuthology.orchestra.run.smithi122.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:32:46.223 INFO:teuthology.orchestra.run.smithi122.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:32:46.223 INFO:teuthology.orchestra.run.smithi122.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:32:46.223 INFO:teuthology.orchestra.run.smithi122.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:32:46.223 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:46.223 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:32:46.477 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-fuse 2024-03-25T16:32:47.043 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:04:29 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:32:47.194 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:32:47.195 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:32:47.195 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-25T16:32:47.195 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:32:47.195 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:32:47.195 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:32:47.195 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-25T16:32:47.195 INFO:teuthology.orchestra.run.smithi122.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:32:47.195 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:47.195 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:32:47.195 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:32:47.195 INFO:teuthology.orchestra.run.smithi122.stdout:Install 2 Packages 2024-03-25T16:32:47.196 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:47.196 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 964 k 2024-03-25T16:32:47.196 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 2.8 M 2024-03-25T16:32:47.196 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:32:47.328 INFO:teuthology.orchestra.run.smithi122.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 596 kB/s | 79 kB 00:00 2024-03-25T16:32:50.578 INFO:teuthology.orchestra.run.smithi122.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 262 kB/s | 885 kB 00:03 2024-03-25T16:32:50.580 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:32:50.580 INFO:teuthology.orchestra.run.smithi122.stdout:Total 285 kB/s | 964 kB 00:03 2024-03-25T16:32:50.581 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:32:50.603 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:32:50.603 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:32:50.678 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:32:50.679 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:32:50.856 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:32:51.020 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:32:51.086 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:32:51.732 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:32:51.732 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:32:52.036 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:32:52.036 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:52.036 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:32:52.036 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:32:52.036 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:52.037 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:32:52.376 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-volume 2024-03-25T16:32:52.743 INFO:teuthology.orchestra.run.smithi204.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 462 kB/s | 17 MB 00:37 2024-03-25T16:32:52.747 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:32:52.747 INFO:teuthology.orchestra.run.smithi204.stdout:Total 725 kB/s | 30 MB 00:41 2024-03-25T16:32:52.770 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:32:52.811 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:32:52.811 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:32:52.950 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:04:34 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:32:53.108 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:32:53.108 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:32:53.108 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repository Size 2024-03-25T16:32:53.108 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:32:53.108 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:32:53.109 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:32:53.109 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-25T16:32:53.109 INFO:teuthology.orchestra.run.smithi122.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:32:53.109 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:53.109 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:32:53.109 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:32:53.109 INFO:teuthology.orchestra.run.smithi122.stdout:Install 2 Packages 2024-03-25T16:32:53.109 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:53.110 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 332 k 2024-03-25T16:32:53.110 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 1.5 M 2024-03-25T16:32:53.110 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:32:53.230 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:32:53.231 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:32:53.235 INFO:teuthology.orchestra.run.smithi122.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 551 kB/s | 69 kB 00:00 2024-03-25T16:32:54.135 INFO:teuthology.orchestra.run.smithi122.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 257 kB/s | 264 kB 00:01 2024-03-25T16:32:54.136 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:32:54.136 INFO:teuthology.orchestra.run.smithi122.stdout:Total 323 kB/s | 332 kB 00:01 2024-03-25T16:32:54.137 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:32:54.145 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:32:54.145 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:32:54.182 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:32:54.255 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:32:54.333 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:32:54.517 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:32:54.558 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:32:54.562 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:32:54.578 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:32:54.578 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:32:54.579 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:32:54.579 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:54.612 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:32:54.654 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:32:54.681 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:32:54.752 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:32:54.797 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:32:54.830 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:32:54.921 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:32:54.999 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:32:55.041 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:32:55.113 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:32:55.167 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:32:55.205 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:32:55.262 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:32:55.326 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:32:55.326 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:32:55.511 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:32:55.559 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:32:55.645 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:32:55.645 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:32:55.659 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:32:55.659 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:55.659 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:32:55.659 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:32:55.659 INFO:teuthology.orchestra.run.smithi122.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:32:55.659 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:55.659 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:32:55.671 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:32:55.747 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:32:55.782 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:32:55.815 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:32:55.847 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:32:55.882 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:32:55.925 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:32:56.007 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados-devel 2024-03-25T16:32:56.014 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:32:56.056 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:32:56.128 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:32:56.265 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:32:56.391 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:32:56.557 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:32:56.584 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:04:38 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:32:56.586 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:32:56.612 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:32:56.612 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:32:56.612 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:32:56.612 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:32:56.612 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:32:56.612 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:32:56.734 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:32:56.735 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:32:56.735 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T16:32:56.736 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:32:56.736 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:32:56.736 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:32:56.736 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:56.736 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:32:56.736 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:32:56.736 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T16:32:56.736 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:56.736 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 127 k 2024-03-25T16:32:56.736 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 456 k 2024-03-25T16:32:56.736 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:32:57.549 INFO:teuthology.orchestra.run.smithi122.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 156 kB/s | 127 kB 00:00 2024-03-25T16:32:57.549 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:32:57.550 INFO:teuthology.orchestra.run.smithi122.stdout:Total 155 kB/s | 127 kB 00:00 2024-03-25T16:32:57.550 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:32:57.556 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:32:57.556 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:32:57.612 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:32:57.613 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:32:57.755 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:32:57.856 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:32:58.096 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:32:58.124 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:32:58.124 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:32:58.124 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:32:58.124 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:32:58.124 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:32:58.125 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:32:58.293 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:32:58.564 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:32:58.587 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:32:58.587 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:32:58.587 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:32:58.587 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:32:58.588 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:32:58.588 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:32:58.601 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:32:58.601 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:58.601 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:32:58.601 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:58.602 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:32:58.602 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:32:58.789 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:32:58.812 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:32:58.812 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:32:58.812 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:32:58.812 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:32:58.812 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:32:58.813 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:32:58.876 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs2 2024-03-25T16:32:58.891 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:32:59.442 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:04:41 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:32:59.539 INFO:teuthology.orchestra.run.smithi122.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:32:59.597 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:32:59.598 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-25T16:32:59.598 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:32:59.673 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs-devel 2024-03-25T16:33:00.240 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:04:42 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:33:00.390 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:33:00.391 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:33:00.391 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T16:33:00.391 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:33:00.391 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:33:00.391 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:33:00.391 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:00.391 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:33:00.391 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:33:00.391 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T16:33:00.392 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:00.392 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 31 k 2024-03-25T16:33:00.392 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 140 k 2024-03-25T16:33:00.392 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:33:00.693 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:33:00.693 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:33:00.693 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:33:00.693 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:33:00.694 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:33:00.694 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:33:00.694 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:33:00.694 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:33:00.694 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:33:00.694 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:33:00.694 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:33:00.694 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:33:00.694 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:33:00.694 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:33:00.694 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:33:00.694 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:33:00.694 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:33:00.694 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:33:00.695 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:33:00.695 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:33:00.695 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:33:00.695 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:33:00.695 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:33:00.695 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:33:00.695 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:33:00.695 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:33:00.695 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:33:00.695 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:33:00.696 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:33:00.696 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:33:00.696 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:33:00.697 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:33:00.697 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:33:00.697 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:33:00.697 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:33:00.697 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:33:01.019 INFO:teuthology.orchestra.run.smithi122.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 50 kB/s | 31 kB 00:00 2024-03-25T16:33:01.020 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:01.020 INFO:teuthology.orchestra.run.smithi122.stdout:Total 50 kB/s | 31 kB 00:00 2024-03-25T16:33:01.020 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:33:01.024 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:33:01.024 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:33:01.048 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:33:01.048 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:33:01.184 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:33:01.302 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:33:01.661 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:33:01.661 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:01.661 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:33:01.661 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:01.661 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:01.661 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:01.661 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:33:01.661 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:01.662 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:01.662 INFO:teuthology.orchestra.run.smithi204.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:33:01.662 INFO:teuthology.orchestra.run.smithi204.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:33:01.662 INFO:teuthology.orchestra.run.smithi204.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:33:01.662 INFO:teuthology.orchestra.run.smithi204.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:33:01.662 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:33:01.662 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:33:01.662 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:33:01.662 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:33:01.662 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:33:01.662 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:33:01.662 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:33:01.662 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:33:01.662 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:33:01.663 INFO:teuthology.orchestra.run.smithi204.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:33:01.663 INFO:teuthology.orchestra.run.smithi204.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:33:01.663 INFO:teuthology.orchestra.run.smithi204.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:33:01.663 INFO:teuthology.orchestra.run.smithi204.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:33:01.663 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:33:01.663 INFO:teuthology.orchestra.run.smithi204.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:33:01.663 INFO:teuthology.orchestra.run.smithi204.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:33:01.663 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:33:01.663 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:33:01.663 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:33:01.663 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:33:01.663 INFO:teuthology.orchestra.run.smithi204.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:33:01.663 INFO:teuthology.orchestra.run.smithi204.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:33:01.663 INFO:teuthology.orchestra.run.smithi204.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:33:01.664 INFO:teuthology.orchestra.run.smithi204.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:33:01.664 INFO:teuthology.orchestra.run.smithi204.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:33:01.664 INFO:teuthology.orchestra.run.smithi204.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:33:01.664 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:01.664 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:33:01.668 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:33:01.934 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:33:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:01.935 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:33:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:01.935 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:01.935 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:33:02.053 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-base 2024-03-25T16:33:02.095 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados2 2024-03-25T16:33:02.619 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:04:40 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:33:02.669 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:04:44 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:33:02.717 INFO:teuthology.orchestra.run.smithi204.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:33:02.769 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:33:02.770 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-25T16:33:02.770 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:33:02.771 INFO:teuthology.orchestra.run.smithi122.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:33:02.827 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:33:02.827 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-25T16:33:02.827 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:33:02.844 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install cephadm 2024-03-25T16:33:02.887 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librbd1 2024-03-25T16:33:03.407 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:04:41 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:33:03.482 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:04:45 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:33:03.556 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:33:03.557 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:33:03.557 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-25T16:33:03.557 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:33:03.557 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:33:03.558 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:33:03.558 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:03.558 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:33:03.558 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:33:03.558 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T16:33:03.558 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:03.558 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 747 k 2024-03-25T16:33:03.558 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 754 k 2024-03-25T16:33:03.558 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:33:03.584 INFO:teuthology.orchestra.run.smithi122.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:33:03.638 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:33:03.639 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-25T16:33:03.639 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:33:03.690 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rados 2024-03-25T16:33:04.257 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:04:46 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:33:04.352 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:33:04.405 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:33:04.405 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-25T16:33:04.405 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:33:04.454 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rgw 2024-03-25T16:33:05.018 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:04:47 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:33:05.116 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:33:05.167 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:33:05.168 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-25T16:33:05.168 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:33:05.220 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-cephfs 2024-03-25T16:33:05.250 INFO:teuthology.orchestra.run.smithi204.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 442 kB/s | 747 kB 00:01 2024-03-25T16:33:05.251 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:05.252 INFO:teuthology.orchestra.run.smithi204.stdout:Total 441 kB/s | 747 kB 00:01 2024-03-25T16:33:05.252 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:33:05.258 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:33:05.258 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:33:05.264 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:33:05.264 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:33:05.339 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:33:05.772 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:33:05.781 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:04:47 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:33:05.856 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:33:05.878 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:33:05.930 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:33:05.931 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-25T16:33:05.931 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:33:05.978 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rbd 2024-03-25T16:33:06.226 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:33:06.512 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:33:06.512 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:06.512 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:33:06.512 INFO:teuthology.orchestra.run.smithi204.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:33:06.512 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:06.512 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:33:06.539 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:04:48 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:33:06.635 INFO:teuthology.orchestra.run.smithi122.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:33:06.686 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:33:06.687 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-25T16:33:06.687 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:33:06.734 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:33:06.740 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-fuse 2024-03-25T16:33:07.302 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:04:49 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:33:07.314 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:04:45 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:33:07.449 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:33:07.450 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:33:07.450 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-03-25T16:33:07.450 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:33:07.450 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:33:07.450 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:33:07.450 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:07.451 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:33:07.451 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:33:07.451 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T16:33:07.451 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:07.451 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 85 k 2024-03-25T16:33:07.451 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 231 k 2024-03-25T16:33:07.451 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:33:07.468 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:33:07.469 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:33:07.469 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T16:33:07.469 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:33:07.469 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:33:07.469 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:33:07.469 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:07.469 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:33:07.470 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:33:07.470 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T16:33:07.470 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:07.470 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 145 k 2024-03-25T16:33:07.470 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 436 k 2024-03-25T16:33:07.470 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:33:08.806 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 63 kB/s | 85 kB 00:01 2024-03-25T16:33:08.806 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:08.807 INFO:teuthology.orchestra.run.smithi122.stdout:Total 63 kB/s | 85 kB 00:01 2024-03-25T16:33:08.807 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:33:08.815 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:33:08.815 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:33:08.875 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:33:08.875 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:33:09.011 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:33:09.065 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 91 kB/s | 145 kB 00:01 2024-03-25T16:33:09.065 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:09.066 INFO:teuthology.orchestra.run.smithi204.stdout:Total 91 kB/s | 145 kB 00:01 2024-03-25T16:33:09.066 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:33:09.075 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:33:09.075 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:33:09.112 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:33:09.131 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:33:09.132 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:33:09.278 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:33:09.332 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:33:09.357 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:33:09.357 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:33:09.357 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:33:09.358 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:09.558 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:33:09.893 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:33:09.893 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:09.893 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:33:09.894 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:09.894 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:09.894 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:33:10.115 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-mirror 2024-03-25T16:33:10.268 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:33:10.268 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:10.268 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:33:10.268 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:10.268 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:10.268 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:33:10.514 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr 2024-03-25T16:33:10.694 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:04:52 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:33:10.840 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:33:10.841 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:33:10.841 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T16:33:10.841 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:33:10.841 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:33:10.841 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:33:10.841 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:10.841 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:33:10.841 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:33:10.841 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T16:33:10.842 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:10.842 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 3.1 M 2024-03-25T16:33:10.842 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 13 M 2024-03-25T16:33:10.842 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:33:11.089 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:04:49 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:33:11.187 INFO:teuthology.orchestra.run.smithi204.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:33:11.238 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:33:11.240 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-25T16:33:11.240 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:33:11.313 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:33:11.874 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:04:49 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:33:12.025 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:33:12.025 INFO:teuthology.orchestra.run.smithi204.stdout:====================================================================================== 2024-03-25T16:33:12.026 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-25T16:33:12.026 INFO:teuthology.orchestra.run.smithi204.stdout:====================================================================================== 2024-03-25T16:33:12.026 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:33:12.026 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:33:12.026 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-25T16:33:12.026 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:33:12.026 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:33:12.026 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:33:12.027 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:33:12.027 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:33:12.027 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:33:12.027 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:33:12.027 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:33:12.027 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:33:12.027 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:33:12.027 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:33:12.027 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:12.027 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:33:12.027 INFO:teuthology.orchestra.run.smithi204.stdout:====================================================================================== 2024-03-25T16:33:12.027 INFO:teuthology.orchestra.run.smithi204.stdout:Install 12 Packages 2024-03-25T16:33:12.028 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:12.028 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 6.9 M 2024-03-25T16:33:12.028 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 26 M 2024-03-25T16:33:12.028 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:33:13.002 INFO:teuthology.orchestra.run.smithi204.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 32 kB/s | 15 kB 00:00 2024-03-25T16:33:13.019 INFO:teuthology.orchestra.run.smithi204.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 64 kB/s | 31 kB 00:00 2024-03-25T16:33:13.178 INFO:teuthology.orchestra.run.smithi204.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.7 MB/s | 1.0 MB 00:00 2024-03-25T16:33:13.361 INFO:teuthology.orchestra.run.smithi204.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 696 kB/s | 238 kB 00:00 2024-03-25T16:33:13.461 INFO:teuthology.orchestra.run.smithi204.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.0 MB/s | 863 kB 00:00 2024-03-25T16:33:13.678 INFO:teuthology.orchestra.run.smithi204.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 90 kB/s | 19 kB 00:00 2024-03-25T16:33:13.770 INFO:teuthology.orchestra.run.smithi204.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.3 MB/s | 548 kB 00:00 2024-03-25T16:33:13.887 INFO:teuthology.orchestra.run.smithi204.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 9.8 MB/s | 2.0 MB 00:00 2024-03-25T16:33:13.912 INFO:teuthology.orchestra.run.smithi204.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.0 MB/s | 144 kB 00:00 2024-03-25T16:33:13.946 INFO:teuthology.orchestra.run.smithi204.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 526 kB/s | 31 kB 00:00 2024-03-25T16:33:13.979 INFO:teuthology.orchestra.run.smithi204.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-25T16:33:20.521 INFO:teuthology.orchestra.run.smithi204.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 228 kB/s | 1.8 MB 00:07 2024-03-25T16:33:20.528 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:20.528 INFO:teuthology.orchestra.run.smithi204.stdout:Total 826 kB/s | 6.9 MB 00:08 2024-03-25T16:33:20.547 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:33:20.567 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:33:20.567 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:33:20.741 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:33:20.742 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:33:21.259 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:33:21.338 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:33:21.382 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:33:21.428 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:33:21.467 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:33:21.564 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:33:21.694 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:33:21.758 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 290 kB/s | 3.1 MB 00:10 2024-03-25T16:33:21.759 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:21.759 INFO:teuthology.orchestra.run.smithi122.stdout:Total 290 kB/s | 3.1 MB 00:10 2024-03-25T16:33:21.760 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:33:21.771 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:33:21.771 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:33:21.786 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:33:21.830 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:33:21.846 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:33:21.847 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:33:21.858 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:33:21.890 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:33:22.134 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:33:22.182 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:33:22.273 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:33:22.342 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:33:22.367 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:33:22.367 INFO:teuthology.orchestra.run.smithi122.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:33:22.367 INFO:teuthology.orchestra.run.smithi122.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:33:22.367 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:33:22.367 INFO:teuthology.orchestra.run.smithi122.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:33:22.367 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:22.917 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:33:22.918 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:33:22.918 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:33:22.918 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:33:22.918 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:33:22.918 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:33:22.918 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:33:22.918 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:33:22.918 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:33:22.918 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:33:22.918 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:33:22.918 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:33:23.275 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:33:23.275 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:23.275 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:33:23.275 INFO:teuthology.orchestra.run.smithi204.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:33:23.275 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:33:23.275 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:33:23.275 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:33:23.275 INFO:teuthology.orchestra.run.smithi204.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:33:23.275 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:33:23.275 INFO:teuthology.orchestra.run.smithi204.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:33:23.275 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:33:23.276 INFO:teuthology.orchestra.run.smithi204.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:33:23.276 INFO:teuthology.orchestra.run.smithi204.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:33:23.276 INFO:teuthology.orchestra.run.smithi204.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:33:23.276 INFO:teuthology.orchestra.run.smithi204.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:33:23.276 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:23.276 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:33:23.482 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:33:23.482 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:23.482 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:33:23.482 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:23.482 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:23.482 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:33:23.525 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:33:23.749 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-nbd 2024-03-25T16:33:24.091 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:05:02 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:33:24.243 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:33:24.244 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================================ 2024-03-25T16:33:24.244 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-25T16:33:24.244 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================================ 2024-03-25T16:33:24.244 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:33:24.244 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:33:24.244 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-25T16:33:24.244 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:33:24.245 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:33:24.245 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:33:24.245 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:33:24.245 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:33:24.245 INFO:teuthology.orchestra.run.smithi204.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:33:24.245 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:33:24.245 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:33:24.245 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:33:24.245 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:33:24.245 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:33:24.245 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:24.245 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:33:24.245 INFO:teuthology.orchestra.run.smithi204.stdout:============================================================================================ 2024-03-25T16:33:24.245 INFO:teuthology.orchestra.run.smithi204.stdout:Install 12 Packages 2024-03-25T16:33:24.246 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:24.246 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 37 M 2024-03-25T16:33:24.246 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 209 M 2024-03-25T16:33:24.246 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:33:24.334 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:05:06 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:33:24.482 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:33:24.482 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:33:24.483 INFO:teuthology.orchestra.run.smithi122.stdout: Package Architecture Version Repository Size 2024-03-25T16:33:24.483 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:33:24.483 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:33:24.483 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:33:24.483 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:24.483 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:33:24.483 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:33:24.483 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T16:33:24.483 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:24.483 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 171 k 2024-03-25T16:33:24.483 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 494 k 2024-03-25T16:33:24.483 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:33:24.820 INFO:teuthology.orchestra.run.smithi204.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 179 kB/s | 32 kB 00:00 2024-03-25T16:33:24.886 INFO:teuthology.orchestra.run.smithi204.stdout:(2/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-03-25T16:33:24.911 INFO:teuthology.orchestra.run.smithi204.stdout:(3/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 191 kB/s | 17 kB 00:00 2024-03-25T16:33:24.937 INFO:teuthology.orchestra.run.smithi204.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 683 kB/s | 34 kB 00:00 2024-03-25T16:33:25.104 INFO:teuthology.orchestra.run.smithi204.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 25 MB/s | 4.7 MB 00:00 2024-03-25T16:33:25.171 INFO:teuthology.orchestra.run.smithi204.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-25T16:33:25.196 INFO:teuthology.orchestra.run.smithi204.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-25T16:33:25.513 INFO:teuthology.orchestra.run.smithi204.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-25T16:33:25.595 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 154 kB/s | 171 kB 00:01 2024-03-25T16:33:25.596 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:25.596 INFO:teuthology.orchestra.run.smithi122.stdout:Total 154 kB/s | 171 kB 00:01 2024-03-25T16:33:25.596 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:33:25.605 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:33:25.605 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:33:25.665 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:33:25.666 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:33:25.818 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:33:25.928 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:33:26.352 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:33:26.371 INFO:teuthology.orchestra.run.smithi204.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 228 kB/s | 195 kB 00:00 2024-03-25T16:33:26.555 INFO:teuthology.orchestra.run.smithi204.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.3 MB/s | 245 kB 00:00 2024-03-25T16:33:26.639 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:33:26.639 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:26.639 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:33:26.639 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:33:26.640 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:26.640 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:33:26.853 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:33:26.955 INFO:teuthology.orchestra.run.smithi204.stdout:(11/12): libgfortran-11.4.1-3.el9.x86_64.rpm 457 kB/s | 803 kB 00:01 2024-03-25T16:33:27.422 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:05:09 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:33:27.575 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:33:27.575 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:33:27.575 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T16:33:27.575 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:33:27.575 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:33:27.575 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:33:27.575 INFO:teuthology.orchestra.run.smithi122.stdout:Installing dependencies: 2024-03-25T16:33:27.576 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:33:27.576 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:27.576 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:33:27.576 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:33:27.576 INFO:teuthology.orchestra.run.smithi122.stdout:Install 2 Packages 2024-03-25T16:33:27.576 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:33:27.576 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 91 M 2024-03-25T16:33:27.577 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 498 M 2024-03-25T16:33:27.577 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:33:43.454 INFO:teuthology.orchestra.run.smithi204.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 402 kB/s | 7.4 MB 00:18 2024-03-25T16:33:43.457 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:43.457 INFO:teuthology.orchestra.run.smithi204.stdout:Total 1.9 MB/s | 37 MB 00:19 2024-03-25T16:33:43.500 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:33:43.526 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:33:43.526 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:33:43.881 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:33:43.881 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:33:44.578 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:33:44.652 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:33:44.691 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:33:44.798 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:33:44.836 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:33:45.015 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:33:45.044 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:33:45.118 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:33:45.773 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:33:45.882 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:33:47.120 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:33:48.138 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:33:48.169 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:33:50.731 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:33:50.731 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:33:50.731 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:33:50.731 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:33:50.731 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:33:50.731 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:33:50.731 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:33:50.731 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:33:50.731 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:33:50.731 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:33:50.732 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:33:50.732 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:33:51.189 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:33:51.190 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:51.190 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:33:51.190 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:33:51.190 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:33:51.190 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:33:51.190 INFO:teuthology.orchestra.run.smithi204.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:33:51.190 INFO:teuthology.orchestra.run.smithi204.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:33:51.190 INFO:teuthology.orchestra.run.smithi204.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:33:51.190 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:33:51.190 INFO:teuthology.orchestra.run.smithi204.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:33:51.190 INFO:teuthology.orchestra.run.smithi204.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:33:51.190 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:33:51.191 INFO:teuthology.orchestra.run.smithi204.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:33:51.191 INFO:teuthology.orchestra.run.smithi204.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:33:51.191 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:51.191 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:33:51.589 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:33:52.163 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:05:30 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:33:52.312 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:33:52.313 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-25T16:33:52.313 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-25T16:33:52.313 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-25T16:33:52.313 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:33:52.313 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:33:52.313 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-25T16:33:52.314 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:33:52.314 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:33:52.314 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:33:52.314 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:33:52.314 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:33:52.314 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:33:52.314 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:33:52.314 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:33:52.314 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:33:52.314 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:33:52.314 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:33:52.314 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:33:52.314 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:52.314 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:33:52.315 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-25T16:33:52.315 INFO:teuthology.orchestra.run.smithi204.stdout:Install 13 Packages 2024-03-25T16:33:52.315 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:52.315 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 2.0 M 2024-03-25T16:33:52.315 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 26 M 2024-03-25T16:33:52.315 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:33:52.587 INFO:teuthology.orchestra.run.smithi204.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 91 kB/s | 18 kB 00:00 2024-03-25T16:33:52.612 INFO:teuthology.orchestra.run.smithi204.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 109 kB/s | 24 kB 00:00 2024-03-25T16:33:52.663 INFO:teuthology.orchestra.run.smithi204.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-25T16:33:52.688 INFO:teuthology.orchestra.run.smithi204.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:33:52.721 INFO:teuthology.orchestra.run.smithi204.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T16:33:52.746 INFO:teuthology.orchestra.run.smithi204.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 731 kB/s | 43 kB 00:00 2024-03-25T16:33:52.864 INFO:teuthology.orchestra.run.smithi204.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-25T16:33:52.889 INFO:teuthology.orchestra.run.smithi204.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 193 kB/s | 32 kB 00:00 2024-03-25T16:33:53.031 INFO:teuthology.orchestra.run.smithi204.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.2 MB/s | 203 kB 00:00 2024-03-25T16:33:53.073 INFO:teuthology.orchestra.run.smithi204.stdout:(10/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 76 kB/s | 52 kB 00:00 2024-03-25T16:33:53.098 INFO:teuthology.orchestra.run.smithi204.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 883 kB/s | 59 kB 00:00 2024-03-25T16:33:53.156 INFO:teuthology.orchestra.run.smithi204.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.8 MB/s | 1.0 MB 00:00 2024-03-25T16:33:53.257 INFO:teuthology.orchestra.run.smithi204.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 488 kB/s | 90 kB 00:00 2024-03-25T16:33:53.259 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:53.259 INFO:teuthology.orchestra.run.smithi204.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-25T16:33:53.259 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:33:53.273 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:33:53.273 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:33:53.367 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:33:53.367 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:33:53.886 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:33:54.066 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:33:54.124 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:33:54.180 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:33:54.217 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:33:54.259 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:33:54.354 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:33:54.462 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:33:54.511 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:33:55.252 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:33:55.307 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:33:55.341 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:33:55.390 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:33:55.411 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:33:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:33:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:33:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:33:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:33:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:33:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:33:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:33:56.180 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:33:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:33:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:33:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:33:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:33:56.181 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:33:56.547 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:33:56.548 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:56.548 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:33:56.548 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:33:56.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:33:56.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:33:56.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:33:56.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:33:56.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:33:56.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:33:56.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:33:56.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:33:56.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:33:56.548 INFO:teuthology.orchestra.run.smithi204.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:33:56.549 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:33:56.549 INFO:teuthology.orchestra.run.smithi204.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:33:56.549 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:56.549 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:33:56.802 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:33:57.372 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:05:35 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:33:57.522 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:33:57.523 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-25T16:33:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-25T16:33:57.523 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-25T16:33:57.523 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:33:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:33:57.523 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-25T16:33:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:33:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:33:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:33:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:33:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:33:57.523 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:57.524 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:33:57.524 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-25T16:33:57.524 INFO:teuthology.orchestra.run.smithi204.stdout:Install 6 Packages 2024-03-25T16:33:57.524 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:33:57.524 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 6.8 M 2024-03-25T16:33:57.524 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 33 M 2024-03-25T16:33:57.524 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:33:58.078 INFO:teuthology.orchestra.run.smithi204.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 2.2 MB/s | 248 kB 00:00 2024-03-25T16:33:58.645 INFO:teuthology.orchestra.run.smithi204.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.0 MB/s | 548 kB 00:00 2024-03-25T16:33:58.745 INFO:teuthology.orchestra.run.smithi204.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 578 kB/s | 58 kB 00:00 2024-03-25T16:33:58.821 INFO:teuthology.orchestra.run.smithi204.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 680 kB/s | 51 kB 00:00 2024-03-25T16:33:59.054 INFO:teuthology.orchestra.run.smithi204.stdout:(5/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 127 kB/s | 138 kB 00:01 2024-03-25T16:33:59.813 INFO:teuthology.orchestra.run.smithi204.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.1 MB/s | 5.8 MB 00:01 2024-03-25T16:33:59.815 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:33:59.815 INFO:teuthology.orchestra.run.smithi204.stdout:Total 3.0 MB/s | 6.8 MB 00:02 2024-03-25T16:33:59.828 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:33:59.838 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:33:59.838 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:33:59.928 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:33:59.929 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:34:00.221 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:34:00.395 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:34:00.456 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:34:00.986 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:34:01.067 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:34:01.128 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:34:01.154 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:34:01.916 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:34:01.916 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:34:01.917 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:34:01.917 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:34:01.917 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:34:01.917 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:34:02.227 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:34:02.228 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:02.228 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:34:02.228 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:34:02.228 INFO:teuthology.orchestra.run.smithi204.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:34:02.228 INFO:teuthology.orchestra.run.smithi204.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:34:02.228 INFO:teuthology.orchestra.run.smithi204.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:34:02.228 INFO:teuthology.orchestra.run.smithi204.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:34:02.228 INFO:teuthology.orchestra.run.smithi204.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:34:02.228 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:02.228 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:34:02.477 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-fuse 2024-03-25T16:34:03.041 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:05:41 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:34:03.190 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:34:03.191 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:03.191 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-25T16:34:03.191 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:03.191 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:34:03.191 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:34:03.191 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-25T16:34:03.191 INFO:teuthology.orchestra.run.smithi204.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:34:03.191 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:03.191 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:34:03.192 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:03.192 INFO:teuthology.orchestra.run.smithi204.stdout:Install 2 Packages 2024-03-25T16:34:03.192 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:03.192 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 964 k 2024-03-25T16:34:03.192 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 2.8 M 2024-03-25T16:34:03.192 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:34:03.323 INFO:teuthology.orchestra.run.smithi204.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 601 kB/s | 79 kB 00:00 2024-03-25T16:34:05.498 INFO:teuthology.orchestra.run.smithi204.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 384 kB/s | 885 kB 00:02 2024-03-25T16:34:05.500 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:05.500 INFO:teuthology.orchestra.run.smithi204.stdout:Total 417 kB/s | 964 kB 00:02 2024-03-25T16:34:05.500 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:34:05.524 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:34:05.524 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:34:05.601 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:34:05.601 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:34:05.786 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:34:05.924 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:34:05.998 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:34:06.660 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:34:06.660 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:34:06.994 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:34:06.994 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:06.994 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:34:06.994 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:34:06.994 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:06.994 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:34:07.260 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-volume 2024-03-25T16:34:07.836 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:05:45 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:34:07.984 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:34:07.985 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:07.985 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repository Size 2024-03-25T16:34:07.985 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:07.986 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:34:07.986 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:34:07.986 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-25T16:34:07.986 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:34:07.986 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:07.986 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:34:07.986 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:07.986 INFO:teuthology.orchestra.run.smithi204.stdout:Install 2 Packages 2024-03-25T16:34:07.986 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:07.986 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 332 k 2024-03-25T16:34:07.986 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 1.5 M 2024-03-25T16:34:07.986 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:34:08.103 INFO:teuthology.orchestra.run.smithi204.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 591 kB/s | 69 kB 00:00 2024-03-25T16:34:09.370 INFO:teuthology.orchestra.run.smithi204.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 191 kB/s | 264 kB 00:01 2024-03-25T16:34:09.371 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:09.371 INFO:teuthology.orchestra.run.smithi204.stdout:Total 240 kB/s | 332 kB 00:01 2024-03-25T16:34:09.372 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:34:09.381 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:34:09.381 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:34:09.431 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:34:09.431 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:34:09.584 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:34:09.759 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:34:09.789 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:34:09.805 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:34:09.805 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:09.805 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:34:09.805 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:10.576 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:34:10.929 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:34:10.930 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:10.930 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:34:10.930 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:34:10.930 INFO:teuthology.orchestra.run.smithi204.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:34:10.930 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:10.930 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:34:11.259 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados-devel 2024-03-25T16:34:11.840 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:05:49 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:34:11.990 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:34:11.991 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:11.991 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T16:34:11.991 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:11.991 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:34:11.991 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:34:11.991 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:11.992 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:34:11.992 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:11.992 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T16:34:11.992 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:11.992 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 127 k 2024-03-25T16:34:11.992 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 456 k 2024-03-25T16:34:11.992 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:34:12.967 INFO:teuthology.orchestra.run.smithi204.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 130 kB/s | 127 kB 00:00 2024-03-25T16:34:12.968 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:12.968 INFO:teuthology.orchestra.run.smithi204.stdout:Total 130 kB/s | 127 kB 00:00 2024-03-25T16:34:12.968 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:34:12.974 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:34:12.974 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:34:13.031 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:34:13.032 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:34:13.176 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:34:13.266 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:13.719 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:14.001 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:14.002 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:14.002 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:34:14.002 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:14.002 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:14.002 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:34:14.245 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install libcephfs2 2024-03-25T16:34:14.821 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:05:52 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:34:14.921 INFO:teuthology.orchestra.run.smithi204.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:14.976 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:34:14.977 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-25T16:34:14.978 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:34:15.052 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install libcephfs-devel 2024-03-25T16:34:15.621 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:05:53 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:34:15.770 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:34:15.771 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:15.771 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T16:34:15.771 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:15.771 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:34:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:34:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:15.772 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:34:15.772 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:15.772 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T16:34:15.772 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:15.772 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 31 k 2024-03-25T16:34:15.772 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 140 k 2024-03-25T16:34:15.772 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:34:16.304 INFO:teuthology.orchestra.run.smithi204.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 58 kB/s | 31 kB 00:00 2024-03-25T16:34:16.305 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:16.305 INFO:teuthology.orchestra.run.smithi204.stdout:Total 58 kB/s | 31 kB 00:00 2024-03-25T16:34:16.305 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:34:16.309 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:34:16.309 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:34:16.337 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:34:16.337 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:34:16.474 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:34:16.586 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:16.940 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:17.208 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:17.209 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:17.209 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:34:17.209 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:17.209 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:17.209 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:34:17.365 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados2 2024-03-25T16:34:17.934 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:05:55 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:34:18.030 INFO:teuthology.orchestra.run.smithi204.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:18.081 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:34:18.082 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-25T16:34:18.083 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:34:18.131 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librbd1 2024-03-25T16:34:18.703 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:05:56 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:34:18.801 INFO:teuthology.orchestra.run.smithi204.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:18.855 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:34:18.856 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-25T16:34:18.856 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:34:18.906 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rados 2024-03-25T16:34:18.944 INFO:teuthology.orchestra.run.smithi122.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 447 kB/s | 22 MB 00:51 2024-03-25T16:34:19.515 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:05:57 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:34:19.617 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:19.672 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:34:19.673 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-25T16:34:19.673 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:34:19.728 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rgw 2024-03-25T16:34:20.312 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:05:58 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:34:20.409 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:20.461 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:34:20.462 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-25T16:34:20.462 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:34:20.516 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-cephfs 2024-03-25T16:34:21.085 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:05:59 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:34:21.182 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:21.235 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:34:21.236 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-25T16:34:21.236 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:34:21.288 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rbd 2024-03-25T16:34:21.855 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:05:59 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:34:21.951 INFO:teuthology.orchestra.run.smithi204.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:34:22.003 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:34:22.004 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-25T16:34:22.004 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:34:22.058 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-fuse 2024-03-25T16:34:22.631 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:06:00 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:34:22.781 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:34:22.782 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:22.782 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-03-25T16:34:22.782 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:22.782 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:34:22.782 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:34:22.782 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:22.783 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:34:22.783 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:22.783 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T16:34:22.783 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:22.783 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 85 k 2024-03-25T16:34:22.783 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 231 k 2024-03-25T16:34:22.783 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:34:23.491 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 120 kB/s | 85 kB 00:00 2024-03-25T16:34:23.492 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:23.492 INFO:teuthology.orchestra.run.smithi204.stdout:Total 120 kB/s | 85 kB 00:00 2024-03-25T16:34:23.492 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:34:23.500 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:34:23.500 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:34:23.556 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:34:23.557 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:34:23.695 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:34:23.823 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:24.218 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:24.481 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:24.481 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:24.481 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:34:24.481 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:24.481 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:24.481 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:34:24.727 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-mirror 2024-03-25T16:34:25.306 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:06:03 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:34:25.463 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:34:25.463 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:25.463 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T16:34:25.463 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:25.463 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:34:25.463 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:34:25.463 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:25.463 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:34:25.463 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:25.464 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T16:34:25.464 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:25.464 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 3.1 M 2024-03-25T16:34:25.464 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 13 M 2024-03-25T16:34:25.464 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:34:38.852 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 236 kB/s | 3.1 MB 00:13 2024-03-25T16:34:38.853 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:38.853 INFO:teuthology.orchestra.run.smithi204.stdout:Total 236 kB/s | 3.1 MB 00:13 2024-03-25T16:34:38.853 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:34:38.864 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:34:38.864 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:34:38.941 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:34:38.942 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:34:39.372 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:34:39.436 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:39.461 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:39.461 INFO:teuthology.orchestra.run.smithi204.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:34:39.461 INFO:teuthology.orchestra.run.smithi204.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:34:39.461 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:34:39.461 INFO:teuthology.orchestra.run.smithi204.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:34:39.461 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:40.488 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:40.488 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:40.488 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:34:40.488 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:40.488 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:40.488 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:34:40.711 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-nbd 2024-03-25T16:34:41.311 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:06:19 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:34:41.461 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:34:41.462 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:41.462 INFO:teuthology.orchestra.run.smithi204.stdout: Package Architecture Version Repository Size 2024-03-25T16:34:41.462 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:41.462 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:34:41.462 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:34:41.462 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:41.462 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:34:41.462 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:41.462 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T16:34:41.462 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:41.462 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 171 k 2024-03-25T16:34:41.462 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 494 k 2024-03-25T16:34:41.462 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:34:42.514 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 163 kB/s | 171 kB 00:01 2024-03-25T16:34:42.515 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:42.515 INFO:teuthology.orchestra.run.smithi204.stdout:Total 163 kB/s | 171 kB 00:01 2024-03-25T16:34:42.515 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:34:42.524 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:34:42.524 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:34:42.583 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:34:42.583 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:34:42.729 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:34:42.839 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:43.269 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:43.549 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:34:43.549 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:43.549 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:34:43.549 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:43.549 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:43.549 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:34:43.757 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:34:44.325 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:06:22 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:34:44.478 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:34:44.478 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:44.478 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T16:34:44.479 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:44.479 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:34:44.479 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:34:44.479 INFO:teuthology.orchestra.run.smithi204.stdout:Installing dependencies: 2024-03-25T16:34:44.479 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:34:44.479 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:44.479 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:34:44.479 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:34:44.479 INFO:teuthology.orchestra.run.smithi204.stdout:Install 2 Packages 2024-03-25T16:34:44.479 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:34:44.480 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 91 M 2024-03-25T16:34:44.480 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 498 M 2024-03-25T16:34:44.480 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T16:36:17.410 INFO:teuthology.orchestra.run.smithi204.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 247 kB/s | 22 MB 01:32 2024-03-25T16:37:14.460 INFO:teuthology.orchestra.run.smithi122.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 308 kB/s | 68 MB 03:46 2024-03-25T16:37:14.461 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:37:14.461 INFO:teuthology.orchestra.run.smithi122.stdout:Total 409 kB/s | 91 MB 03:46 2024-03-25T16:37:14.461 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T16:37:14.466 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T16:37:14.466 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T16:37:15.063 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T16:37:15.063 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T16:37:18.164 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T16:37:25.300 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:37:25.451 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:37:28.928 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:37:28.928 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:37:29.185 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:37:29.185 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:37:29.185 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T16:37:29.185 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:29.185 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:37:29.185 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:37:29.185 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T16:37:29.419 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:37:29.987 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:09:11 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T16:37:30.142 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T16:37:30.143 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:37:30.143 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T16:37:30.143 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:37:30.143 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T16:37:30.144 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:37:30.144 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:37:30.144 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T16:37:30.144 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T16:37:30.144 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T16:37:30.144 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T16:37:30.144 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 416 M 2024-03-25T16:37:30.144 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 1.7 G 2024-03-25T16:37:30.144 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T16:39:17.589 INFO:teuthology.orchestra.run.smithi204.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 255 kB/s | 68 MB 04:33 2024-03-25T16:39:17.590 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:39:17.590 INFO:teuthology.orchestra.run.smithi204.stdout:Total 340 kB/s | 91 MB 04:33 2024-03-25T16:39:17.591 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T16:39:17.595 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T16:39:17.595 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T16:39:18.173 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T16:39:18.173 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T16:39:21.257 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T16:39:28.403 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:39:28.472 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:39:31.984 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:39:31.984 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:39:32.258 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:39:32.259 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:39:32.259 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T16:39:32.259 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:32.259 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:39:32.259 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:39:32.259 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T16:39:32.505 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:39:33.074 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:11:11 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T16:39:33.223 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T16:39:33.224 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:39:33.224 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T16:39:33.224 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:39:33.224 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T16:39:33.224 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:39:33.224 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:39:33.224 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 416 M 2024-03-25T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 1.7 G 2024-03-25T16:39:33.225 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T17:03:06.472 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 277 kB/s | 416 MB 25:36 2024-03-25T17:03:06.473 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:03:06.473 INFO:teuthology.orchestra.run.smithi122.stdout:Total 277 kB/s | 416 MB 25:36 2024-03-25T17:03:06.473 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T17:03:06.476 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T17:03:06.476 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T17:03:08.949 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T17:03:08.949 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T17:03:52.361 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T17:03:52.926 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:04:00.752 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:04:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:04:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:04:01.009 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T17:04:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:04:01.009 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:04:01.009 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T17:04:01.246 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:04:01.814 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:35:43 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T17:04:01.962 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T17:04:01.963 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2024-03-25T17:04:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T17:04:01.963 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2024-03-25T17:04:01.963 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T17:04:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:04:01.963 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:04:01.963 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T17:04:01.963 INFO:teuthology.orchestra.run.smithi122.stdout:======================================================================================== 2024-03-25T17:04:01.964 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T17:04:01.964 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:04:01.964 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 1.4 M 2024-03-25T17:04:01.964 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 6.4 M 2024-03-25T17:04:01.964 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T17:04:06.853 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 299 kB/s | 1.4 MB 00:04 2024-03-25T17:04:06.854 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:04:06.855 INFO:teuthology.orchestra.run.smithi122.stdout:Total 299 kB/s | 1.4 MB 00:04 2024-03-25T17:04:06.855 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T17:04:06.859 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T17:04:06.859 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T17:04:06.902 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T17:04:06.903 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T17:04:07.155 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T17:04:07.259 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:04:07.685 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:04:07.956 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:04:07.957 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:04:07.957 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T17:04:07.957 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:04:07.957 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:04:07.957 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T17:04:08.158 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:04:08.720 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:35:50 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T17:04:08.866 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T17:04:08.867 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:04:08.867 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T17:04:08.867 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:04:08.867 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T17:04:08.867 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:04:08.867 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:04:08.867 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T17:04:08.867 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:04:08.867 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T17:04:08.867 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:04:08.868 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 176 M 2024-03-25T17:04:08.868 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 753 M 2024-03-25T17:04:08.868 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T17:07:55.933 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 250 kB/s | 416 MB 28:22 2024-03-25T17:07:55.933 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:07:55.933 INFO:teuthology.orchestra.run.smithi204.stdout:Total 250 kB/s | 416 MB 28:22 2024-03-25T17:07:55.933 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T17:07:55.937 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T17:07:55.937 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T17:07:58.403 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T17:07:58.404 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T17:08:40.498 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T17:08:40.594 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:08:48.885 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:08:49.146 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:08:49.146 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:08:49.146 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T17:08:49.146 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:08:49.147 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:08:49.147 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T17:08:49.403 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:08:49.971 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:40:27 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T17:08:50.121 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T17:08:50.122 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-25T17:08:50.122 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T17:08:50.122 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-25T17:08:50.122 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T17:08:50.122 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:08:50.122 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:08:50.122 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T17:08:50.122 INFO:teuthology.orchestra.run.smithi204.stdout:======================================================================================== 2024-03-25T17:08:50.123 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T17:08:50.123 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:08:50.123 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 1.4 M 2024-03-25T17:08:50.123 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 6.4 M 2024-03-25T17:08:50.123 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T17:08:56.489 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 230 kB/s | 1.4 MB 00:06 2024-03-25T17:08:56.490 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:08:56.490 INFO:teuthology.orchestra.run.smithi204.stdout:Total 229 kB/s | 1.4 MB 00:06 2024-03-25T17:08:56.490 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T17:08:56.494 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T17:08:56.494 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T17:08:56.540 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T17:08:56.540 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T17:08:56.803 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T17:08:56.904 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:08:57.288 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:08:57.569 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:08:57.569 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:08:57.570 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T17:08:57.570 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:08:57.570 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:08:57.570 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T17:08:57.754 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:08:58.318 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:40:36 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T17:08:58.467 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T17:08:58.467 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:08:58.467 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T17:08:58.467 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:08:58.467 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T17:08:58.467 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:08:58.467 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:08:58.467 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T17:08:58.468 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:08:58.468 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T17:08:58.468 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:08:58.468 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 176 M 2024-03-25T17:08:58.468 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 753 M 2024-03-25T17:08:58.468 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T17:14:33.083 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 288 kB/s | 176 MB 10:24 2024-03-25T17:14:33.083 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:14:33.084 INFO:teuthology.orchestra.run.smithi122.stdout:Total 288 kB/s | 176 MB 10:24 2024-03-25T17:14:33.084 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T17:14:33.087 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T17:14:33.087 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T17:14:34.151 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T17:14:34.151 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T17:14:52.166 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T17:14:52.254 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:14:57.063 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:14:57.336 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:14:57.337 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:14:57.337 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T17:14:57.337 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:14:57.337 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:14:57.337 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T17:14:57.521 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:14:58.080 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 0:46:40 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T17:14:58.231 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T17:14:58.232 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:14:58.232 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T17:14:58.232 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:14:58.232 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T17:14:58.232 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:14:58.233 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:14:58.233 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T17:14:58.233 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:14:58.233 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T17:14:58.233 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:14:58.233 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 672 M 2024-03-25T17:14:58.233 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 3.1 G 2024-03-25T17:14:58.233 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T17:15:31.431 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 458 kB/s | 176 MB 06:32 2024-03-25T17:15:31.432 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:31.432 INFO:teuthology.orchestra.run.smithi204.stdout:Total 458 kB/s | 176 MB 06:32 2024-03-25T17:15:31.432 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T17:15:31.436 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T17:15:31.436 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T17:15:32.498 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T17:15:32.498 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T17:15:50.760 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T17:15:50.861 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:15:55.592 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:15:55.855 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:15:55.855 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:15:55.855 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T17:15:55.855 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:55.855 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:15:55.855 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T17:15:56.071 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:15:56.637 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 0:47:34 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T17:15:56.784 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T17:15:56.784 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:15:56.785 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T17:15:56.785 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:15:56.785 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T17:15:56.785 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:15:56.785 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:15:56.785 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T17:15:56.785 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:15:56.785 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T17:15:56.785 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:15:56.786 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 672 M 2024-03-25T17:15:56.786 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 3.1 G 2024-03-25T17:15:56.786 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T17:36:58.488 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 521 kB/s | 672 MB 22:00 2024-03-25T17:36:58.511 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:36:58.511 INFO:teuthology.orchestra.run.smithi122.stdout:Total 521 kB/s | 672 MB 22:00 2024-03-25T17:36:58.512 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T17:36:58.512 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T17:36:58.512 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T17:37:02.465 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T17:37:02.465 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T17:38:13.368 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T17:38:13.607 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:38:22.890 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:38:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:38:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:38:23.156 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T17:38:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:38:23.156 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:38:23.156 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T17:38:23.424 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:38:23.987 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 1:10:05 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T17:38:24.083 INFO:teuthology.orchestra.run.smithi122.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:38:24.135 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T17:38:24.136 INFO:teuthology.orchestra.run.smithi122.stdout:Nothing to do. 2024-03-25T17:38:24.136 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T17:38:24.211 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:38:24.770 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 1:10:06 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T17:38:24.917 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T17:38:24.918 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:38:24.918 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T17:38:24.918 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:38:24.919 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T17:38:24.919 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:38:24.919 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:38:24.919 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T17:38:24.919 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:38:24.919 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T17:38:24.919 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:38:24.919 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 20 M 2024-03-25T17:38:24.919 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 103 M 2024-03-25T17:38:24.919 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T17:39:11.589 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 432 kB/s | 20 MB 00:46 2024-03-25T17:39:11.590 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:39:11.590 INFO:teuthology.orchestra.run.smithi122.stdout:Total 432 kB/s | 20 MB 00:46 2024-03-25T17:39:11.590 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T17:39:11.594 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T17:39:11.594 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T17:39:11.742 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T17:39:11.743 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T17:39:13.995 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T17:39:14.096 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:39:15.060 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:39:15.317 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:39:15.317 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:39:15.317 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T17:39:15.317 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:39:15.317 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:39:15.317 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T17:39:15.489 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:39:16.053 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 1:10:58 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T17:39:16.200 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T17:39:16.200 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:39:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T17:39:16.201 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:39:16.201 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T17:39:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:39:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:39:16.201 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T17:39:16.201 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:39:16.201 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T17:39:16.201 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:39:16.202 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 48 M 2024-03-25T17:39:16.202 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 173 M 2024-03-25T17:39:16.202 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T17:40:48.384 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 530 kB/s | 48 MB 01:32 2024-03-25T17:40:48.385 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:40:48.385 INFO:teuthology.orchestra.run.smithi122.stdout:Total 530 kB/s | 48 MB 01:32 2024-03-25T17:40:48.385 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T17:40:48.388 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T17:40:48.388 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T17:40:48.700 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T17:40:48.700 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T17:40:53.466 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T17:40:53.548 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:54.979 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:55.246 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:40:55.247 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T17:40:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:40:55.247 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:40:55.247 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T17:40:55.415 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:40:55.977 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 1:12:37 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T17:40:56.124 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T17:40:56.125 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:40:56.125 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T17:40:56.125 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:40:56.125 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T17:40:56.126 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:40:56.126 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:40:56.126 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T17:40:56.126 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:40:56.126 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T17:40:56.126 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:40:56.126 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 99 M 2024-03-25T17:40:56.126 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 449 M 2024-03-25T17:40:56.126 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T17:44:10.867 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 518 kB/s | 99 MB 03:14 2024-03-25T17:44:10.867 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:44:10.867 INFO:teuthology.orchestra.run.smithi122.stdout:Total 518 kB/s | 99 MB 03:14 2024-03-25T17:44:10.867 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T17:44:10.870 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T17:44:10.870 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T17:44:11.478 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T17:44:11.479 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T17:44:21.715 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T17:44:21.822 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:24.847 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:25.112 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:25.112 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:44:25.112 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T17:44:25.112 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:44:25.112 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:44:25.112 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T17:44:25.297 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:44:25.857 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 1:16:07 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T17:44:26.005 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T17:44:26.006 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:44:26.006 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T17:44:26.006 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:44:26.007 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T17:44:26.007 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:44:26.007 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:44:26.007 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T17:44:26.007 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:44:26.007 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T17:44:26.007 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:44:26.007 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 287 M 2024-03-25T17:44:26.007 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 1.2 G 2024-03-25T17:44:26.007 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T17:44:41.609 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 399 kB/s | 672 MB 28:44 2024-03-25T17:44:41.610 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:44:41.610 INFO:teuthology.orchestra.run.smithi204.stdout:Total 399 kB/s | 672 MB 28:44 2024-03-25T17:44:41.610 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T17:44:41.615 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T17:44:41.615 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T17:44:45.591 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T17:44:45.592 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T17:45:56.456 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T17:45:56.561 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:46:06.729 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:46:06.991 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:46:06.991 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:46:06.991 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T17:46:06.991 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:46:06.992 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:46:06.992 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T17:46:07.292 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:46:07.863 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 1:17:45 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T17:46:07.959 INFO:teuthology.orchestra.run.smithi204.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:46:08.011 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T17:46:08.012 INFO:teuthology.orchestra.run.smithi204.stdout:Nothing to do. 2024-03-25T17:46:08.012 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T17:46:08.082 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:46:08.655 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 1:17:46 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T17:46:08.803 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T17:46:08.804 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:46:08.804 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T17:46:08.804 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:46:08.804 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T17:46:08.804 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:46:08.805 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:46:08.805 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T17:46:08.805 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:46:08.805 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T17:46:08.805 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:46:08.805 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 20 M 2024-03-25T17:46:08.805 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 103 M 2024-03-25T17:46:08.805 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T17:47:20.917 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 280 kB/s | 20 MB 01:12 2024-03-25T17:47:20.917 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:47:20.918 INFO:teuthology.orchestra.run.smithi204.stdout:Total 280 kB/s | 20 MB 01:12 2024-03-25T17:47:20.918 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T17:47:20.921 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T17:47:20.921 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T17:47:21.072 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T17:47:21.073 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T17:47:23.304 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T17:47:23.490 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:47:24.397 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:47:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:47:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:47:24.665 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T17:47:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:47:24.665 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:47:24.665 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T17:47:24.878 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:47:25.440 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 1:19:03 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T17:47:25.588 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T17:47:25.588 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:47:25.588 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T17:47:25.588 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:47:25.588 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T17:47:25.588 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:47:25.588 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:47:25.589 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T17:47:25.589 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:47:25.589 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T17:47:25.589 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:47:25.589 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 48 M 2024-03-25T17:47:25.589 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 173 M 2024-03-25T17:47:25.589 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T17:49:39.822 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 364 kB/s | 48 MB 02:14 2024-03-25T17:49:39.823 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:39.823 INFO:teuthology.orchestra.run.smithi204.stdout:Total 364 kB/s | 48 MB 02:14 2024-03-25T17:49:39.823 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T17:49:39.826 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T17:49:39.826 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T17:49:40.145 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T17:49:40.145 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T17:49:44.925 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T17:49:45.010 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:46.336 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:46.592 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:46.592 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:49:46.592 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T17:49:46.592 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:46.592 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:49:46.592 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T17:49:46.783 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:49:47.348 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 1:21:25 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T17:49:47.497 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T17:49:47.497 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:49:47.498 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T17:49:47.498 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:49:47.498 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T17:49:47.498 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:49:47.498 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:49:47.498 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T17:49:47.498 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:49:47.498 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T17:49:47.498 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:49:47.498 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 99 M 2024-03-25T17:49:47.498 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 449 M 2024-03-25T17:49:47.498 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T17:54:53.282 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 330 kB/s | 99 MB 05:05 2024-03-25T17:54:53.283 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:53.283 INFO:teuthology.orchestra.run.smithi204.stdout:Total 330 kB/s | 99 MB 05:05 2024-03-25T17:54:53.283 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T17:54:53.286 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T17:54:53.286 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T17:54:53.893 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T17:54:53.894 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T17:55:04.155 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T17:55:04.245 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:07.229 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:07.500 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:55:07.500 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:55:07.500 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T17:55:07.500 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:07.500 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:55:07.500 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T17:55:07.691 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:55:08.258 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 1:26:46 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T17:55:08.406 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T17:55:08.407 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:55:08.407 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T17:55:08.407 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:55:08.407 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T17:55:08.407 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:55:08.407 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:55:08.407 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T17:55:08.408 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T17:55:08.408 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T17:55:08.408 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T17:55:08.408 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 287 M 2024-03-25T17:55:08.408 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 1.2 G 2024-03-25T17:55:08.408 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T17:57:20.280 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 379 kB/s | 287 MB 12:54 2024-03-25T17:57:20.280 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:20.280 INFO:teuthology.orchestra.run.smithi122.stdout:Total 379 kB/s | 287 MB 12:54 2024-03-25T17:57:20.280 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T17:57:20.283 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T17:57:20.283 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T17:57:21.996 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T17:57:21.997 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T17:57:51.229 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T17:57:51.341 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:57:59.174 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:57:59.463 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:57:59.463 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:57:59.463 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T17:57:59.463 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:57:59.463 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:57:59.463 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T17:57:59.680 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:58:00.245 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 1:29:42 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T17:58:00.393 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T17:58:00.393 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:58:00.394 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T17:58:00.394 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:58:00.394 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T17:58:00.394 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:58:00.394 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:58:00.394 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T17:58:00.394 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:58:00.394 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T17:58:00.394 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:58:00.394 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 5.9 M 2024-03-25T17:58:00.394 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 28 M 2024-03-25T17:58:00.394 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T17:58:17.469 INFO:teuthology.orchestra.run.smithi122.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 356 kB/s | 5.9 MB 00:17 2024-03-25T17:58:17.470 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:17.470 INFO:teuthology.orchestra.run.smithi122.stdout:Total 356 kB/s | 5.9 MB 00:17 2024-03-25T17:58:17.470 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T17:58:17.474 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T17:58:17.474 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T17:58:17.544 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T17:58:17.545 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T17:58:18.271 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T17:58:18.384 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:18.988 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:58:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:58:19.265 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T17:58:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:19.265 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:58:19.265 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T17:58:19.454 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:58:20.012 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 1:30:02 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T17:58:20.161 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T17:58:20.161 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:58:20.161 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T17:58:20.162 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:58:20.162 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T17:58:20.162 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:58:20.162 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:58:20.162 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T17:58:20.162 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:58:20.162 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T17:58:20.162 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:58:20.162 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 109 k 2024-03-25T17:58:20.162 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 440 k 2024-03-25T17:58:20.162 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T17:58:21.047 INFO:teuthology.orchestra.run.smithi122.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 123 kB/s | 109 kB 00:00 2024-03-25T17:58:21.047 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:21.048 INFO:teuthology.orchestra.run.smithi122.stdout:Total 123 kB/s | 109 kB 00:00 2024-03-25T17:58:21.048 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T17:58:21.051 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T17:58:21.051 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T17:58:21.087 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T17:58:21.087 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T17:58:21.202 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T17:58:21.295 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:58:21.680 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:58:21.947 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:58:21.947 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:58:21.948 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T17:58:21.948 INFO:teuthology.orchestra.run.smithi122.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:21.948 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:58:21.948 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T17:58:22.128 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:58:22.689 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 1:30:04 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T17:58:22.838 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T17:58:22.839 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:58:22.839 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T17:58:22.839 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:58:22.839 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T17:58:22.839 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:58:22.839 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:58:22.839 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T17:58:22.839 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:58:22.839 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T17:58:22.840 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:58:22.840 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 5.4 M 2024-03-25T17:58:22.840 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 25 M 2024-03-25T17:58:22.840 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T17:58:35.532 INFO:teuthology.orchestra.run.smithi122.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 432 kB/s | 5.4 MB 00:12 2024-03-25T17:58:35.533 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:35.533 INFO:teuthology.orchestra.run.smithi122.stdout:Total 432 kB/s | 5.4 MB 00:12 2024-03-25T17:58:35.534 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T17:58:35.538 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T17:58:35.538 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T17:58:35.604 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T17:58:35.605 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T17:58:36.272 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T17:58:36.380 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:36.878 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:37.145 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:37.145 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:58:37.145 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T17:58:37.145 INFO:teuthology.orchestra.run.smithi122.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:37.145 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:58:37.146 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T17:58:37.345 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librados2-debuginfo 2024-03-25T17:58:37.909 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 1:30:19 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T17:58:38.058 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T17:58:38.059 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:58:38.059 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T17:58:38.059 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:58:38.059 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T17:58:38.059 INFO:teuthology.orchestra.run.smithi122.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:58:38.059 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:58:38.060 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T17:58:38.060 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T17:58:38.060 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T17:58:38.060 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T17:58:38.060 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 44 M 2024-03-25T17:58:38.060 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 253 M 2024-03-25T17:58:38.060 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T18:01:14.646 INFO:teuthology.orchestra.run.smithi122.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 290 kB/s | 44 MB 02:36 2024-03-25T18:01:14.647 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:14.647 INFO:teuthology.orchestra.run.smithi122.stdout:Total 290 kB/s | 44 MB 02:36 2024-03-25T18:01:14.647 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T18:01:14.650 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T18:01:14.650 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T18:01:14.947 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T18:01:14.947 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T18:01:20.000 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T18:01:20.086 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:01:22.002 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:01:22.275 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:01:22.275 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:01:22.275 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T18:01:22.275 INFO:teuthology.orchestra.run.smithi122.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:22.275 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:01:22.275 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T18:01:22.461 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:01:23.021 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 1:33:05 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T18:01:23.170 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T18:01:23.171 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:01:23.171 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T18:01:23.171 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:01:23.171 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T18:01:23.171 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:01:23.171 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:01:23.171 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T18:01:23.171 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:01:23.171 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T18:01:23.171 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:01:23.171 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 52 M 2024-03-25T18:01:23.171 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 308 M 2024-03-25T18:01:23.172 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T18:04:18.183 INFO:teuthology.orchestra.run.smithi122.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 303 kB/s | 52 MB 02:55 2024-03-25T18:04:18.184 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:18.184 INFO:teuthology.orchestra.run.smithi122.stdout:Total 303 kB/s | 52 MB 02:55 2024-03-25T18:04:18.184 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T18:04:18.185 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T18:04:18.185 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T18:04:18.527 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T18:04:18.527 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T18:04:24.497 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T18:04:24.888 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:27.085 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:27.370 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:27.370 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:27.370 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T18:04:27.370 INFO:teuthology.orchestra.run.smithi122.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:27.370 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:27.371 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T18:04:27.608 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:04:28.166 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 1:36:10 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T18:04:28.316 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T18:04:28.317 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:28.317 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T18:04:28.317 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:28.317 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T18:04:28.317 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:04:28.317 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:28.317 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T18:04:28.318 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:28.318 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T18:04:28.318 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:28.318 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 615 k 2024-03-25T18:04:28.318 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 2.1 M 2024-03-25T18:04:28.318 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T18:04:31.199 INFO:teuthology.orchestra.run.smithi122.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 213 kB/s | 615 kB 00:02 2024-03-25T18:04:31.200 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:31.200 INFO:teuthology.orchestra.run.smithi122.stdout:Total 213 kB/s | 615 kB 00:02 2024-03-25T18:04:31.201 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T18:04:31.205 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T18:04:31.205 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T18:04:31.245 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T18:04:31.245 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T18:04:31.414 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T18:04:31.515 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:31.898 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi122.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:32.159 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T18:04:32.346 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:04:32.904 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 1:36:14 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T18:04:33.053 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T18:04:33.053 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:33.054 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T18:04:33.054 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:33.054 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T18:04:33.054 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:04:33.054 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:33.054 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T18:04:33.054 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:33.054 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T18:04:33.054 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:33.055 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 771 k 2024-03-25T18:04:33.055 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 2.7 M 2024-03-25T18:04:33.055 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T18:04:35.536 INFO:teuthology.orchestra.run.smithi122.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 311 kB/s | 771 kB 00:02 2024-03-25T18:04:35.536 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:35.537 INFO:teuthology.orchestra.run.smithi122.stdout:Total 311 kB/s | 771 kB 00:02 2024-03-25T18:04:35.537 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T18:04:35.541 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T18:04:35.541 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T18:04:35.582 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T18:04:35.582 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T18:04:35.764 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T18:04:35.867 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:04:36.229 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:04:36.492 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:04:36.492 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:36.492 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T18:04:36.492 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:36.492 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:36.492 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T18:04:36.708 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:04:37.264 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 1:36:19 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T18:04:37.413 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T18:04:37.413 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:37.414 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T18:04:37.414 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:37.414 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T18:04:37.414 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:04:37.414 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:37.414 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T18:04:37.414 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:37.414 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T18:04:37.414 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:37.414 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 1.1 M 2024-03-25T18:04:37.414 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 4.3 M 2024-03-25T18:04:37.415 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T18:04:41.364 INFO:teuthology.orchestra.run.smithi122.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 291 kB/s | 1.1 MB 00:03 2024-03-25T18:04:41.365 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:41.365 INFO:teuthology.orchestra.run.smithi122.stdout:Total 291 kB/s | 1.1 MB 00:03 2024-03-25T18:04:41.366 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T18:04:41.369 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T18:04:41.369 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T18:04:41.412 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T18:04:41.412 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T18:04:41.633 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T18:04:41.747 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:42.149 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:42.418 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:42.418 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:42.418 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T18:04:42.418 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:42.418 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:42.418 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T18:04:42.630 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:04:43.196 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 1:36:25 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T18:04:43.346 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T18:04:43.346 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:43.346 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T18:04:43.347 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:43.347 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T18:04:43.347 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:04:43.347 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:43.347 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T18:04:43.347 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:43.347 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T18:04:43.347 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:43.347 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 337 k 2024-03-25T18:04:43.347 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 1.0 M 2024-03-25T18:04:43.347 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T18:04:45.484 INFO:teuthology.orchestra.run.smithi122.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 158 kB/s | 337 kB 00:02 2024-03-25T18:04:45.484 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:45.485 INFO:teuthology.orchestra.run.smithi122.stdout:Total 158 kB/s | 337 kB 00:02 2024-03-25T18:04:45.485 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T18:04:45.489 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T18:04:45.489 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T18:04:45.528 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T18:04:45.529 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T18:04:45.671 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T18:04:45.776 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:46.140 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:46.433 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:04:46.433 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:46.433 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T18:04:46.433 INFO:teuthology.orchestra.run.smithi122.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:46.433 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:46.433 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T18:04:46.641 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:04:47.202 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 1:36:29 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T18:04:47.354 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T18:04:47.354 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:47.354 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T18:04:47.354 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:47.354 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T18:04:47.354 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:04:47.354 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:47.354 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T18:04:47.354 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:47.354 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T18:04:47.354 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:47.355 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 509 k 2024-03-25T18:04:47.355 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 2.5 M 2024-03-25T18:04:47.355 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T18:04:50.330 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 171 kB/s | 509 kB 00:02 2024-03-25T18:04:50.331 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:50.331 INFO:teuthology.orchestra.run.smithi122.stdout:Total 171 kB/s | 509 kB 00:02 2024-03-25T18:04:50.332 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T18:04:50.336 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T18:04:50.336 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T18:04:50.376 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T18:04:50.376 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T18:04:50.542 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T18:04:50.639 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:51.083 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:51.350 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:51.350 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:51.350 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T18:04:51.350 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:51.350 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:51.350 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T18:04:51.539 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:04:52.102 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 1:36:34 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T18:04:52.250 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T18:04:52.251 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:52.251 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T18:04:52.251 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:52.251 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T18:04:52.251 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:04:52.251 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:52.251 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T18:04:52.251 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:04:52.251 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T18:04:52.251 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:04:52.252 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 52 M 2024-03-25T18:04:52.252 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 298 M 2024-03-25T18:04:52.252 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T18:05:58.510 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 452 kB/s | 287 MB 10:50 2024-03-25T18:05:58.510 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:58.510 INFO:teuthology.orchestra.run.smithi204.stdout:Total 452 kB/s | 287 MB 10:50 2024-03-25T18:05:58.511 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T18:05:58.514 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T18:05:58.514 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T18:06:00.223 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T18:06:00.223 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T18:06:29.663 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T18:06:30.324 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:06:32.338 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:06:32.626 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:06:32.626 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:06:32.626 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T18:06:32.626 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:32.627 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:06:32.627 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T18:06:32.909 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:06:33.482 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 1:38:11 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T18:06:33.631 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T18:06:33.632 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:06:33.632 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T18:06:33.632 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:06:33.632 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T18:06:33.632 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:06:33.632 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:06:33.632 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T18:06:33.633 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:06:33.633 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T18:06:33.633 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:06:33.633 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 5.9 M 2024-03-25T18:06:33.633 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 28 M 2024-03-25T18:06:33.633 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T18:06:49.272 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 451 kB/s | 52 MB 01:57 2024-03-25T18:06:49.273 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:49.273 INFO:teuthology.orchestra.run.smithi122.stdout:Total 451 kB/s | 52 MB 01:57 2024-03-25T18:06:49.273 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T18:06:49.276 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T18:06:49.276 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T18:06:49.609 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T18:06:49.609 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T18:06:54.485 INFO:teuthology.orchestra.run.smithi204.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 292 kB/s | 5.9 MB 00:20 2024-03-25T18:06:54.486 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:54.486 INFO:teuthology.orchestra.run.smithi204.stdout:Total 292 kB/s | 5.9 MB 00:20 2024-03-25T18:06:54.486 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T18:06:54.490 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T18:06:54.490 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T18:06:54.562 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T18:06:54.562 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T18:06:55.290 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T18:06:55.442 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:06:55.510 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T18:06:55.603 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:06:55.979 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:06:56.257 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:06:56.257 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:06:56.257 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T18:06:56.257 INFO:teuthology.orchestra.run.smithi204.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:56.257 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:06:56.257 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T18:06:56.469 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:06:57.033 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 1:38:35 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T18:06:57.183 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T18:06:57.184 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:06:57.184 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T18:06:57.184 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:06:57.184 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T18:06:57.184 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:06:57.184 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:06:57.184 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T18:06:57.184 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:06:57.184 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T18:06:57.184 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:06:57.185 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 109 k 2024-03-25T18:06:57.185 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 440 k 2024-03-25T18:06:57.185 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T18:06:57.688 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:06:57.929 INFO:teuthology.orchestra.run.smithi204.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 146 kB/s | 109 kB 00:00 2024-03-25T18:06:57.930 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:57.930 INFO:teuthology.orchestra.run.smithi204.stdout:Total 146 kB/s | 109 kB 00:00 2024-03-25T18:06:57.930 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T18:06:57.934 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T18:06:57.934 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T18:06:57.958 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:06:57.958 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:06:57.958 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T18:06:57.958 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:57.958 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:06:57.958 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T18:06:57.971 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T18:06:57.971 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T18:06:58.085 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T18:06:58.145 DEBUG:teuthology.orchestra.run.smithi122:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:06:58.187 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:06:58.584 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:06:58.704 INFO:teuthology.orchestra.run.smithi122.stdout:Last metadata expiration check: 1:38:40 ago on Mon 25 Mar 2024 04:28:18 PM UTC. 2024-03-25T18:06:58.852 INFO:teuthology.orchestra.run.smithi122.stdout:Dependencies resolved. 2024-03-25T18:06:58.853 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:06:58.853 INFO:teuthology.orchestra.run.smithi122.stdout: Package Arch Version Repo Size 2024-03-25T18:06:58.854 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:06:58.854 INFO:teuthology.orchestra.run.smithi122.stdout:Installing: 2024-03-25T18:06:58.854 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:06:58.854 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:06:58.854 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction Summary 2024-03-25T18:06:58.854 INFO:teuthology.orchestra.run.smithi122.stdout:================================================================================ 2024-03-25T18:06:58.854 INFO:teuthology.orchestra.run.smithi122.stdout:Install 1 Package 2024-03-25T18:06:58.854 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:06:58.854 INFO:teuthology.orchestra.run.smithi122.stdout:Total download size: 1.3 M 2024-03-25T18:06:58.854 INFO:teuthology.orchestra.run.smithi122.stdout:Installed size: 5.5 M 2024-03-25T18:06:58.854 INFO:teuthology.orchestra.run.smithi122.stdout:Downloading Packages: 2024-03-25T18:06:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:06:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:06:58.862 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T18:06:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:58.862 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:06:58.863 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T18:06:59.061 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:06:59.628 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 1:38:37 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T18:06:59.780 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T18:06:59.780 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:06:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T18:06:59.781 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:06:59.781 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T18:06:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:06:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:06:59.781 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T18:06:59.781 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:06:59.781 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T18:06:59.781 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:06:59.781 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 5.4 M 2024-03-25T18:06:59.781 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 25 M 2024-03-25T18:06:59.781 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T18:07:04.031 INFO:teuthology.orchestra.run.smithi122.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 248 kB/s | 1.3 MB 00:05 2024-03-25T18:07:04.032 INFO:teuthology.orchestra.run.smithi122.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:04.033 INFO:teuthology.orchestra.run.smithi122.stdout:Total 248 kB/s | 1.3 MB 00:05 2024-03-25T18:07:04.033 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction check 2024-03-25T18:07:04.037 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction check succeeded. 2024-03-25T18:07:04.037 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction test 2024-03-25T18:07:04.081 INFO:teuthology.orchestra.run.smithi122.stdout:Transaction test succeeded. 2024-03-25T18:07:04.081 INFO:teuthology.orchestra.run.smithi122.stdout:Running transaction 2024-03-25T18:07:04.318 INFO:teuthology.orchestra.run.smithi122.stdout: Preparing : 1/1 2024-03-25T18:07:04.423 INFO:teuthology.orchestra.run.smithi122.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:04.868 INFO:teuthology.orchestra.run.smithi122.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:05.183 INFO:teuthology.orchestra.run.smithi122.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:05.183 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:07:05.183 INFO:teuthology.orchestra.run.smithi122.stdout:Installed: 2024-03-25T18:07:05.183 INFO:teuthology.orchestra.run.smithi122.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:05.183 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:07:05.183 INFO:teuthology.orchestra.run.smithi122.stdout:Complete! 2024-03-25T18:07:05.387 DEBUG:teuthology.parallel:result is None 2024-03-25T18:07:14.823 INFO:teuthology.orchestra.run.smithi204.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 364 kB/s | 5.4 MB 00:15 2024-03-25T18:07:14.824 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:14.824 INFO:teuthology.orchestra.run.smithi204.stdout:Total 364 kB/s | 5.4 MB 00:15 2024-03-25T18:07:14.824 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T18:07:14.828 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T18:07:14.829 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T18:07:14.897 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T18:07:14.897 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T18:07:15.572 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T18:07:15.680 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:07:16.220 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:07:16.488 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:07:16.488 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:07:16.488 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T18:07:16.488 INFO:teuthology.orchestra.run.smithi204.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:16.488 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:07:16.488 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T18:07:16.706 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librados2-debuginfo 2024-03-25T18:07:17.276 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 1:38:55 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T18:07:17.424 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T18:07:17.425 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:07:17.425 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T18:07:17.425 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:07:17.425 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T18:07:17.425 INFO:teuthology.orchestra.run.smithi204.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:07:17.425 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:07:17.426 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T18:07:17.426 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:07:17.426 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T18:07:17.426 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:07:17.426 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 44 M 2024-03-25T18:07:17.426 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 253 M 2024-03-25T18:07:17.426 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T18:09:54.836 INFO:teuthology.orchestra.run.smithi204.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 288 kB/s | 44 MB 02:37 2024-03-25T18:09:54.836 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:54.836 INFO:teuthology.orchestra.run.smithi204.stdout:Total 288 kB/s | 44 MB 02:37 2024-03-25T18:09:54.837 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T18:09:54.839 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T18:09:54.839 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T18:09:55.139 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T18:09:55.139 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T18:10:00.202 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T18:10:00.302 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:10:02.219 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:10:02.490 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:10:02.490 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:10:02.490 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T18:10:02.490 INFO:teuthology.orchestra.run.smithi204.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:02.490 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:10:02.490 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T18:10:02.709 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:10:03.278 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 1:41:41 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T18:10:03.430 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T18:10:03.431 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:10:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T18:10:03.431 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:10:03.431 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T18:10:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:10:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:10:03.431 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T18:10:03.431 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:10:03.431 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T18:10:03.431 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:10:03.432 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 52 M 2024-03-25T18:10:03.432 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 308 M 2024-03-25T18:10:03.432 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T18:12:01.343 INFO:teuthology.orchestra.run.smithi204.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 450 kB/s | 52 MB 01:57 2024-03-25T18:12:01.343 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:12:01.343 INFO:teuthology.orchestra.run.smithi204.stdout:Total 450 kB/s | 52 MB 01:57 2024-03-25T18:12:01.343 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T18:12:01.346 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T18:12:01.346 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T18:12:01.692 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T18:12:01.692 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T18:12:07.698 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T18:12:07.915 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:12:10.212 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:12:10.483 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:12:10.483 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:10.483 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T18:12:10.483 INFO:teuthology.orchestra.run.smithi204.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:12:10.483 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:10.483 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T18:12:10.678 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:12:11.238 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 1:43:49 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T18:12:11.386 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T18:12:11.387 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:11.387 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T18:12:11.387 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:11.387 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T18:12:11.387 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:12:11.387 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:11.387 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T18:12:11.387 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:11.388 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T18:12:11.388 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:11.388 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 615 k 2024-03-25T18:12:11.388 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 2.1 M 2024-03-25T18:12:11.388 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T18:12:13.394 INFO:teuthology.orchestra.run.smithi204.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 306 kB/s | 615 kB 00:02 2024-03-25T18:12:13.395 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:12:13.395 INFO:teuthology.orchestra.run.smithi204.stdout:Total 306 kB/s | 615 kB 00:02 2024-03-25T18:12:13.396 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T18:12:13.399 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T18:12:13.400 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T18:12:13.441 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T18:12:13.441 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T18:12:13.614 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T18:12:13.760 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:12:14.168 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:12:14.445 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:12:14.445 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:14.445 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T18:12:14.445 INFO:teuthology.orchestra.run.smithi204.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:12:14.445 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:14.446 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T18:12:14.658 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:12:15.221 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 1:43:53 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T18:12:15.369 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T18:12:15.370 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:15.370 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T18:12:15.370 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:15.370 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T18:12:15.370 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:12:15.370 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:15.370 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T18:12:15.370 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:15.370 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T18:12:15.371 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:15.371 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 771 k 2024-03-25T18:12:15.371 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 2.7 M 2024-03-25T18:12:15.371 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T18:12:19.589 INFO:teuthology.orchestra.run.smithi204.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 183 kB/s | 771 kB 00:04 2024-03-25T18:12:19.589 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:12:19.590 INFO:teuthology.orchestra.run.smithi204.stdout:Total 183 kB/s | 771 kB 00:04 2024-03-25T18:12:19.590 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T18:12:19.594 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T18:12:19.594 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T18:12:19.636 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T18:12:19.636 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T18:12:19.818 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T18:12:19.920 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:12:20.289 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:12:20.563 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:12:20.563 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:20.563 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T18:12:20.563 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:12:20.563 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:20.564 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T18:12:20.786 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:12:21.352 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 1:43:59 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T18:12:21.501 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T18:12:21.502 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:21.502 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T18:12:21.502 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:21.502 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T18:12:21.502 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:12:21.502 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:21.502 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T18:12:21.502 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:21.502 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T18:12:21.502 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:21.503 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 1.1 M 2024-03-25T18:12:21.503 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 4.3 M 2024-03-25T18:12:21.503 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T18:12:26.534 INFO:teuthology.orchestra.run.smithi204.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 229 kB/s | 1.1 MB 00:05 2024-03-25T18:12:26.534 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:12:26.535 INFO:teuthology.orchestra.run.smithi204.stdout:Total 229 kB/s | 1.1 MB 00:05 2024-03-25T18:12:26.535 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T18:12:26.539 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T18:12:26.539 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T18:12:26.583 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T18:12:26.583 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T18:12:26.806 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T18:12:26.899 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:12:27.299 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:12:27.578 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:12:27.578 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:27.578 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T18:12:27.578 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:12:27.578 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:27.578 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T18:12:27.796 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:12:28.361 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 1:44:06 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T18:12:28.510 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T18:12:28.511 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:28.511 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T18:12:28.511 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:28.511 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T18:12:28.511 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:12:28.511 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:28.511 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T18:12:28.511 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:28.511 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T18:12:28.511 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:28.512 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 337 k 2024-03-25T18:12:28.512 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 1.0 M 2024-03-25T18:12:28.512 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T18:12:30.178 INFO:teuthology.orchestra.run.smithi204.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 202 kB/s | 337 kB 00:01 2024-03-25T18:12:30.179 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:12:30.179 INFO:teuthology.orchestra.run.smithi204.stdout:Total 202 kB/s | 337 kB 00:01 2024-03-25T18:12:30.179 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T18:12:30.183 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T18:12:30.183 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T18:12:30.224 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T18:12:30.225 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T18:12:30.368 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T18:12:30.478 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:12:30.828 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:12:31.094 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:12:31.094 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:31.094 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T18:12:31.094 INFO:teuthology.orchestra.run.smithi204.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:12:31.094 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:31.094 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T18:12:31.320 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:12:31.883 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 1:44:09 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T18:12:32.032 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T18:12:32.033 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:32.033 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T18:12:32.033 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:32.033 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T18:12:32.033 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:12:32.033 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:32.033 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T18:12:32.033 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:32.033 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T18:12:32.034 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:32.034 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 509 k 2024-03-25T18:12:32.034 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 2.5 M 2024-03-25T18:12:32.034 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T18:12:34.781 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 185 kB/s | 509 kB 00:02 2024-03-25T18:12:34.782 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:12:34.783 INFO:teuthology.orchestra.run.smithi204.stdout:Total 185 kB/s | 509 kB 00:02 2024-03-25T18:12:34.783 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T18:12:34.787 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T18:12:34.787 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T18:12:34.826 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T18:12:34.826 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T18:12:35.004 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T18:12:35.113 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:12:35.498 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:12:35.758 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:12:35.758 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:35.758 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T18:12:35.758 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:12:35.758 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:35.758 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T18:12:35.966 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:12:36.527 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 1:44:14 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T18:12:36.675 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T18:12:36.676 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:36.676 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T18:12:36.676 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:36.676 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T18:12:36.676 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:12:36.676 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:36.676 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T18:12:36.676 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:12:36.676 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T18:12:36.676 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:12:36.676 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 52 M 2024-03-25T18:12:36.676 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 298 M 2024-03-25T18:12:36.676 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T18:15:16.293 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 331 kB/s | 52 MB 02:39 2024-03-25T18:15:16.293 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:16.293 INFO:teuthology.orchestra.run.smithi204.stdout:Total 331 kB/s | 52 MB 02:39 2024-03-25T18:15:16.294 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T18:15:16.296 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T18:15:16.297 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T18:15:16.633 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T18:15:16.633 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T18:15:22.552 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T18:15:22.908 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:15:25.063 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:15:25.337 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:15:25.337 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:25.337 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T18:15:25.337 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:25.337 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:25.337 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T18:15:25.533 DEBUG:teuthology.orchestra.run.smithi204:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:15:26.106 INFO:teuthology.orchestra.run.smithi204.stdout:Last metadata expiration check: 1:47:04 ago on Mon 25 Mar 2024 04:28:22 PM UTC. 2024-03-25T18:15:26.256 INFO:teuthology.orchestra.run.smithi204.stdout:Dependencies resolved. 2024-03-25T18:15:26.256 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:15:26.257 INFO:teuthology.orchestra.run.smithi204.stdout: Package Arch Version Repo Size 2024-03-25T18:15:26.257 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:15:26.257 INFO:teuthology.orchestra.run.smithi204.stdout:Installing: 2024-03-25T18:15:26.257 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:15:26.257 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:26.257 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction Summary 2024-03-25T18:15:26.257 INFO:teuthology.orchestra.run.smithi204.stdout:================================================================================ 2024-03-25T18:15:26.257 INFO:teuthology.orchestra.run.smithi204.stdout:Install 1 Package 2024-03-25T18:15:26.257 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:26.257 INFO:teuthology.orchestra.run.smithi204.stdout:Total download size: 1.3 M 2024-03-25T18:15:26.257 INFO:teuthology.orchestra.run.smithi204.stdout:Installed size: 5.5 M 2024-03-25T18:15:26.257 INFO:teuthology.orchestra.run.smithi204.stdout:Downloading Packages: 2024-03-25T18:15:28.744 INFO:teuthology.orchestra.run.smithi204.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 516 kB/s | 1.3 MB 00:02 2024-03-25T18:15:28.745 INFO:teuthology.orchestra.run.smithi204.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:28.745 INFO:teuthology.orchestra.run.smithi204.stdout:Total 515 kB/s | 1.3 MB 00:02 2024-03-25T18:15:28.745 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction check 2024-03-25T18:15:28.750 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction check succeeded. 2024-03-25T18:15:28.750 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction test 2024-03-25T18:15:28.794 INFO:teuthology.orchestra.run.smithi204.stdout:Transaction test succeeded. 2024-03-25T18:15:28.795 INFO:teuthology.orchestra.run.smithi204.stdout:Running transaction 2024-03-25T18:15:29.084 INFO:teuthology.orchestra.run.smithi204.stdout: Preparing : 1/1 2024-03-25T18:15:29.177 INFO:teuthology.orchestra.run.smithi204.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:15:29.618 INFO:teuthology.orchestra.run.smithi204.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:15:29.886 INFO:teuthology.orchestra.run.smithi204.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:15:29.886 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:29.886 INFO:teuthology.orchestra.run.smithi204.stdout:Installed: 2024-03-25T18:15:29.886 INFO:teuthology.orchestra.run.smithi204.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:29.886 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:29.887 INFO:teuthology.orchestra.run.smithi204.stdout:Complete! 2024-03-25T18:15:30.099 DEBUG:teuthology.parallel:result is None 2024-03-25T18:15:30.099 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:15:30.316 DEBUG:teuthology.orchestra.run.smithi122:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:15:30.340 INFO:teuthology.orchestra.run.smithi122.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:15:30.341 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:15:30.341 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:15:30.342 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:15:30.553 DEBUG:teuthology.orchestra.run.smithi204:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:15:30.575 INFO:teuthology.orchestra.run.smithi204.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:15:30.575 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:15:30.575 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:15:30.576 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:15:30.576 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:15:30.577 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:15:30.608 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T18:15:30.608 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:15:30.644 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:15:30.644 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:15:30.644 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:15:30.678 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:15:30.747 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T18:15:30.747 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:15:30.776 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:15:30.844 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:15:30.844 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:15:30.844 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:15:30.874 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:15:30.941 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T18:15:30.941 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:15:30.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:15:31.036 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:15:31.036 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:15:31.036 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:15:31.063 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:15:31.130 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T18:15:31.130 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:15:31.157 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:15:31.224 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:15:31.306 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:15:31.306 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 777 /var/log/ceph 2024-03-25T18:15:31.309 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 777 /var/log/ceph 2024-03-25T18:15:31.335 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:15:31.336 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:15:31.375 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:15:31.402 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:15:31.402 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:15:31.442 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:15:31.528 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:15:31.529 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:15:31.529 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621268', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621268', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi122.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGd7eAzSBHm5d/7BTcOsLLUMmH7tiZ52/vuXEdLczaKP5XPiYpF/sMjLY0JGojUMgBC7U39UxBfDgCB94CgY6dc=', 'smithi204.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEF5xawqKZ/fdvHRzou19KJf/TWLawveGfsRh6QIGO3NEYljtxhlVih68qNJP/Tfjrq49W/+x4/ZkKNnGSgRMmA='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:15:31.529 DEBUG:teuthology.orchestra.run.smithi122:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:15:31.531 DEBUG:teuthology.orchestra.run.smithi204:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:15:31.546 DEBUG:teuthology.orchestra.run.smithi122:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:15:31.587 DEBUG:teuthology.orchestra.run.smithi204:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:15:31.613 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:15:31.613 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:15:31.665 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:15:31.665 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_1 2024-03-25T18:15:31.720 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:15:31.720 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:15:31.720 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T18:15:31.720 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:15:31.720 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:15:31.720 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-25 16:33:22.618350649 +0000 2024-03-25T18:15:31.720 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-25 16:25:05.929772737 +0000 2024-03-25T18:15:31.721 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-25 16:25:05.929772737 +0000 2024-03-25T18:15:31.721 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-25 16:25:05.929772737 +0000 2024-03-25T18:15:31.721 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:15:31.784 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-25T18:15:31.785 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-25T18:15:31.785 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000237961 s, 2.2 MB/s 2024-03-25T18:15:31.786 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:15:31.841 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_2 2024-03-25T18:15:31.896 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:15:31.896 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:15:31.896 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T18:15:31.896 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:15:31.896 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:15:31.896 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-25 16:33:22.619350624 +0000 2024-03-25T18:15:31.897 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-25 16:25:06.273764517 +0000 2024-03-25T18:15:31.897 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-25 16:25:06.273764517 +0000 2024-03-25T18:15:31.897 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-25 16:25:06.273764517 +0000 2024-03-25T18:15:31.897 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:15:31.962 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-25T18:15:31.962 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-25T18:15:31.962 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000180225 s, 2.8 MB/s 2024-03-25T18:15:31.963 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:15:32.018 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_3 2024-03-25T18:15:32.072 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:15:32.072 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:15:32.072 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T18:15:32.072 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:15:32.072 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:15:32.072 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-25 16:33:22.619350624 +0000 2024-03-25T18:15:32.072 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-25 16:25:06.597756775 +0000 2024-03-25T18:15:32.072 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-25 16:25:06.597756775 +0000 2024-03-25T18:15:32.072 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-25 16:25:06.597756775 +0000 2024-03-25T18:15:32.072 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:15:32.137 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-25T18:15:32.137 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-25T18:15:32.137 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000187616 s, 2.7 MB/s 2024-03-25T18:15:32.139 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:15:32.194 DEBUG:teuthology.orchestra.run.smithi122:> stat /dev/vg_nvme/lv_4 2024-03-25T18:15:32.249 INFO:teuthology.orchestra.run.smithi122.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:15:32.249 INFO:teuthology.orchestra.run.smithi122.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:15:32.249 INFO:teuthology.orchestra.run.smithi122.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T18:15:32.250 INFO:teuthology.orchestra.run.smithi122.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:15:32.250 INFO:teuthology.orchestra.run.smithi122.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:15:32.250 INFO:teuthology.orchestra.run.smithi122.stdout:Access: 2024-03-25 16:33:22.619350624 +0000 2024-03-25T18:15:32.250 INFO:teuthology.orchestra.run.smithi122.stdout:Modify: 2024-03-25 16:25:06.926748914 +0000 2024-03-25T18:15:32.250 INFO:teuthology.orchestra.run.smithi122.stdout:Change: 2024-03-25 16:25:06.926748914 +0000 2024-03-25T18:15:32.250 INFO:teuthology.orchestra.run.smithi122.stdout: Birth: 2024-03-25 16:25:06.926748914 +0000 2024-03-25T18:15:32.250 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:15:32.315 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records in 2024-03-25T18:15:32.315 INFO:teuthology.orchestra.run.smithi122.stderr:1+0 records out 2024-03-25T18:15:32.315 INFO:teuthology.orchestra.run.smithi122.stderr:512 bytes copied, 0.000233483 s, 2.2 MB/s 2024-03-25T18:15:32.316 DEBUG:teuthology.orchestra.run.smithi122:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:15:32.372 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:32.372 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T18:15:32.372 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:15:32.389 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:15:32.389 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_1 2024-03-25T18:15:32.445 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:15:32.445 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:15:32.445 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T18:15:32.446 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:15:32.446 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:15:32.446 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-25 16:34:39.685287292 +0000 2024-03-25T18:15:32.446 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-25 16:27:27.940807488 +0000 2024-03-25T18:15:32.446 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-25 16:27:27.940807488 +0000 2024-03-25T18:15:32.446 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-25 16:27:27.940807488 +0000 2024-03-25T18:15:32.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:15:32.512 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-25T18:15:32.512 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-25T18:15:32.512 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000267734 s, 1.9 MB/s 2024-03-25T18:15:32.513 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:15:32.570 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_2 2024-03-25T18:15:32.625 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:15:32.626 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:15:32.626 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T18:15:32.626 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:15:32.626 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:15:32.626 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-25 16:34:39.685287292 +0000 2024-03-25T18:15:32.626 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-25 16:27:28.272802475 +0000 2024-03-25T18:15:32.626 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-25 16:27:28.272802475 +0000 2024-03-25T18:15:32.626 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-25 16:27:28.272802475 +0000 2024-03-25T18:15:32.626 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:15:32.692 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-25T18:15:32.692 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-25T18:15:32.692 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000195954 s, 2.6 MB/s 2024-03-25T18:15:32.693 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:15:32.749 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_3 2024-03-25T18:15:32.804 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:15:32.804 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:15:32.804 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T18:15:32.805 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:15:32.805 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:15:32.805 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-25 16:34:39.686287277 +0000 2024-03-25T18:15:32.805 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-25 16:27:28.615797297 +0000 2024-03-25T18:15:32.805 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-25 16:27:28.615797297 +0000 2024-03-25T18:15:32.805 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-25 16:27:28.615797297 +0000 2024-03-25T18:15:32.805 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:15:32.870 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-25T18:15:32.870 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-25T18:15:32.870 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000179034 s, 2.9 MB/s 2024-03-25T18:15:32.871 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:15:32.927 DEBUG:teuthology.orchestra.run.smithi204:> stat /dev/vg_nvme/lv_4 2024-03-25T18:15:32.982 INFO:teuthology.orchestra.run.smithi204.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:15:32.982 INFO:teuthology.orchestra.run.smithi204.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:15:32.982 INFO:teuthology.orchestra.run.smithi204.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T18:15:32.982 INFO:teuthology.orchestra.run.smithi204.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:15:32.982 INFO:teuthology.orchestra.run.smithi204.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:15:32.982 INFO:teuthology.orchestra.run.smithi204.stdout:Access: 2024-03-25 16:34:39.686287277 +0000 2024-03-25T18:15:32.982 INFO:teuthology.orchestra.run.smithi204.stdout:Modify: 2024-03-25 16:27:28.958792119 +0000 2024-03-25T18:15:32.982 INFO:teuthology.orchestra.run.smithi204.stdout:Change: 2024-03-25 16:27:28.958792119 +0000 2024-03-25T18:15:32.982 INFO:teuthology.orchestra.run.smithi204.stdout: Birth: 2024-03-25 16:27:28.958792119 +0000 2024-03-25T18:15:32.983 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:15:33.048 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records in 2024-03-25T18:15:33.048 INFO:teuthology.orchestra.run.smithi204.stderr:1+0 records out 2024-03-25T18:15:33.048 INFO:teuthology.orchestra.run.smithi204.stderr:512 bytes copied, 0.000200168 s, 2.6 MB/s 2024-03-25T18:15:33.049 DEBUG:teuthology.orchestra.run.smithi204:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:15:33.105 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:33.105 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:15:33.105 INFO:tasks.ceph:Generating config... 2024-03-25T18:15:33.106 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:15:33.106 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:15:33.106 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:15:33.106 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:15:33.106 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:15:33.106 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:15:33.106 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:15:33.106 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:15:33.107 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:15:33.107 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:15:33.107 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:15:33.107 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:15:33.107 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:15:33.107 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:15:33.107 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:15:33.107 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:15:33.107 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:15:33.107 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:15:33.107 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:15:33.107 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:15:33.107 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:15:33.107 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:15:33.107 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:15:33.107 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:15:33.107 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:15:33.108 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:15:33.108 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:15:33.108 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:15:33.108 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:15:33.108 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:15:33.108 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:15:33.108 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:15:33.108 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:15:33.108 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:15:33.108 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:15:33.108 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:15:33.108 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:15:33.108 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:15:33.108 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:15:33.108 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:15:33.108 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:15:33.109 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:15:33.109 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:15:33.109 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:15:33.109 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:15:33.109 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:15:33.109 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:15:33.109 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:15:33.109 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:15:33.109 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:15:33.109 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:15:33.109 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:15:33.109 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:15:33.109 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:15:33.110 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:15:33.161 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:15:33.163 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:15:33.209 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:15:33.235 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.122:6789]'), ('mon.c', '[v1:172.21.15.122:6790]'), ('mon.b', '[v1:172.21.15.204:6789]')] 2024-03-25T18:15:33.236 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.122:6789],[v1:172.21.15.122:6790],[v1:172.21.15.204:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:15:33.237 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:15:33.237 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:15:33.291 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.122:6789]' --addv c '[v1:172.21.15.122:6790]' --addv b '[v1:172.21.15.204:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:33.375 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:33.375 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: generated fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:15:33.375 INFO:teuthology.orchestra.run.smithi122.stdout:setting min_mon_release = quincy 2024-03-25T18:15:33.375 INFO:teuthology.orchestra.run.smithi122.stdout:epoch 0 2024-03-25T18:15:33.376 INFO:teuthology.orchestra.run.smithi122.stdout:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:15:33.376 INFO:teuthology.orchestra.run.smithi122.stdout:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:15:33.376 INFO:teuthology.orchestra.run.smithi122.stdout:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:15:33.376 INFO:teuthology.orchestra.run.smithi122.stdout:min_mon_release 17 (quincy) 2024-03-25T18:15:33.376 INFO:teuthology.orchestra.run.smithi122.stdout:election_strategy: 3 2024-03-25T18:15:33.376 INFO:teuthology.orchestra.run.smithi122.stdout:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:15:33.376 INFO:teuthology.orchestra.run.smithi122.stdout:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:15:33.376 INFO:teuthology.orchestra.run.smithi122.stdout:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:15:33.376 INFO:teuthology.orchestra.run.smithi122.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:15:33.377 DEBUG:teuthology.orchestra.run.smithi122:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:15:33.391 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID b4b19611-696c-4b9f-a4ec-3ee2f6191d98... 2024-03-25T18:15:33.392 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:15:33.434 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:15:33.482 INFO:teuthology.orchestra.run.smithi122.stdout:[global] 2024-03-25T18:15:33.482 INFO:teuthology.orchestra.run.smithi122.stdout: chdir = "" 2024-03-25T18:15:33.482 INFO:teuthology.orchestra.run.smithi122.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:15:33.482 INFO:teuthology.orchestra.run.smithi122.stdout: auth supported = cephx 2024-03-25T18:15:33.482 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.482 INFO:teuthology.orchestra.run.smithi122.stdout: filestore xattr use omap = true 2024-03-25T18:15:33.482 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.482 INFO:teuthology.orchestra.run.smithi122.stdout: mon clock drift allowed = 1.000 2024-03-25T18:15:33.482 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.482 INFO:teuthology.orchestra.run.smithi122.stdout: osd crush chooseleaf type = 0 2024-03-25T18:15:33.482 INFO:teuthology.orchestra.run.smithi122.stdout: auth debug = true 2024-03-25T18:15:33.483 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.483 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on old message = true 2024-03-25T18:15:33.483 INFO:teuthology.orchestra.run.smithi122.stdout: ms die on bug = true 2024-03-25T18:15:33.483 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.483 INFO:teuthology.orchestra.run.smithi122.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:15:33.483 INFO:teuthology.orchestra.run.smithi122.stdout: mon pg warn max object skew = 0 2024-03-25T18:15:33.483 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.483 INFO:teuthology.orchestra.run.smithi122.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:15:33.483 INFO:teuthology.orchestra.run.smithi122.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:15:33.483 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.483 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default size = 2 2024-03-25T18:15:33.483 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.483 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow primary affinity = true 2024-03-25T18:15:33.483 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd allow pg remap = true 2024-03-25T18:15:33.483 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:15:33.484 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:15:33.484 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on no sortbitwise = false 2024-03-25T18:15:33.484 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:15:33.484 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on too few osds = false 2024-03-25T18:15:33.484 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:15:33.484 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:15:33.484 INFO:teuthology.orchestra.run.smithi122.stdout: mon_allow_pool_size_one = true 2024-03-25T18:15:33.484 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.484 INFO:teuthology.orchestra.run.smithi122.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:15:33.484 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.484 INFO:teuthology.orchestra.run.smithi122.stdout: osd default data pool replay window = 5 2024-03-25T18:15:33.484 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.484 INFO:teuthology.orchestra.run.smithi122.stdout: mon allow pool delete = true 2024-03-25T18:15:33.484 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.484 INFO:teuthology.orchestra.run.smithi122.stdout: mon cluster log file level = debug 2024-03-25T18:15:33.484 INFO:teuthology.orchestra.run.smithi122.stdout: debug asserts on shutdown = true 2024-03-25T18:15:33.485 INFO:teuthology.orchestra.run.smithi122.stdout: mon health detail to clog = false 2024-03-25T18:15:33.485 INFO:teuthology.orchestra.run.smithi122.stdout: mon host = "[v1:172.21.15.122:6789],[v1:172.21.15.122:6790],[v1:172.21.15.204:6789]" 2024-03-25T18:15:33.485 INFO:teuthology.orchestra.run.smithi122.stdout: mon client directed command retry = 5 2024-03-25T18:15:33.485 INFO:teuthology.orchestra.run.smithi122.stdout: mon election default strategy = 3 2024-03-25T18:15:33.485 INFO:teuthology.orchestra.run.smithi122.stdout: ms bind msgr2 = False 2024-03-25T18:15:33.485 INFO:teuthology.orchestra.run.smithi122.stdout: ms inject socket failures = 5000 2024-03-25T18:15:33.485 INFO:teuthology.orchestra.run.smithi122.stdout: ms type = async 2024-03-25T18:15:33.485 INFO:teuthology.orchestra.run.smithi122.stdout: osd heartbeat grace = 80 2024-03-25T18:15:33.485 INFO:teuthology.orchestra.run.smithi122.stdout: fsid = b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:15:33.485 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.485 INFO:teuthology.orchestra.run.smithi122.stdout:[osd] 2024-03-25T18:15:33.486 INFO:teuthology.orchestra.run.smithi122.stdout: osd journal size = 100 2024-03-25T18:15:33.486 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.486 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub load threshold = 5.0 2024-03-25T18:15:33.486 INFO:teuthology.orchestra.run.smithi122.stdout: osd scrub max interval = 600 2024-03-25T18:15:33.486 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:15:33.486 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.486 INFO:teuthology.orchestra.run.smithi122.stdout: osd recover clone overlap = true 2024-03-25T18:15:33.486 INFO:teuthology.orchestra.run.smithi122.stdout: osd recovery max chunk = 1048576 2024-03-25T18:15:33.486 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.486 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug shutdown = true 2024-03-25T18:15:33.486 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug op order = true 2024-03-25T18:15:33.486 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify stray on activate = true 2024-03-25T18:15:33.486 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug trim objects = true 2024-03-25T18:15:33.486 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.486 INFO:teuthology.orchestra.run.smithi122.stdout: osd open classes on start = true 2024-03-25T18:15:33.487 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug pg log writeout = true 2024-03-25T18:15:33.487 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.487 INFO:teuthology.orchestra.run.smithi122.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:15:33.487 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.487 INFO:teuthology.orchestra.run.smithi122.stdout: osd map max advance = 10 2024-03-25T18:15:33.487 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.487 INFO:teuthology.orchestra.run.smithi122.stdout: journal zero on create = true 2024-03-25T18:15:33.487 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.487 INFO:teuthology.orchestra.run.smithi122.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:15:33.487 INFO:teuthology.orchestra.run.smithi122.stdout: filestore apply finisher threads = 3 2024-03-25T18:15:33.487 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.487 INFO:teuthology.orchestra.run.smithi122.stdout: bdev debug aio = true 2024-03-25T18:15:33.487 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug misdirected ops = true 2024-03-25T18:15:33.487 INFO:teuthology.orchestra.run.smithi122.stdout: bdev async discard = True 2024-03-25T18:15:33.487 INFO:teuthology.orchestra.run.smithi122.stdout: bdev enable discard = True 2024-03-25T18:15:33.487 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore allocator = stupid 2024-03-25T18:15:33.488 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore block size = 96636764160 2024-03-25T18:15:33.488 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore fsck on mount = True 2024-03-25T18:15:33.488 INFO:teuthology.orchestra.run.smithi122.stdout: bluestore zero block detection = True 2024-03-25T18:15:33.488 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluefs = 1 2024-03-25T18:15:33.488 INFO:teuthology.orchestra.run.smithi122.stdout: debug bluestore = 1 2024-03-25T18:15:33.488 INFO:teuthology.orchestra.run.smithi122.stdout: debug monc = 20 2024-03-25T18:15:33.488 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-25T18:15:33.488 INFO:teuthology.orchestra.run.smithi122.stdout: debug osd = 20 2024-03-25T18:15:33.488 INFO:teuthology.orchestra.run.smithi122.stdout: debug rocksdb = 10 2024-03-25T18:15:33.488 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:15:33.488 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd full ratio = 0.9 2024-03-25T18:15:33.488 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:15:33.488 INFO:teuthology.orchestra.run.smithi122.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:15:33.488 INFO:teuthology.orchestra.run.smithi122.stdout: osd class default list = * 2024-03-25T18:15:33.488 INFO:teuthology.orchestra.run.smithi122.stdout: osd class load list = * 2024-03-25T18:15:33.489 INFO:teuthology.orchestra.run.smithi122.stdout: osd client watch timeout = 120 2024-03-25T18:15:33.489 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify cached snaps = True 2024-03-25T18:15:33.489 INFO:teuthology.orchestra.run.smithi122.stdout: osd debug verify missing on start = True 2024-03-25T18:15:33.489 INFO:teuthology.orchestra.run.smithi122.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:15:33.489 INFO:teuthology.orchestra.run.smithi122.stdout: osd fast shutdown = False 2024-03-25T18:15:33.489 INFO:teuthology.orchestra.run.smithi122.stdout: osd max markdown count = 1000 2024-03-25T18:15:33.489 INFO:teuthology.orchestra.run.smithi122.stdout: osd mclock override recovery settings = True 2024-03-25T18:15:33.489 INFO:teuthology.orchestra.run.smithi122.stdout: osd objectstore = bluestore 2024-03-25T18:15:33.489 INFO:teuthology.orchestra.run.smithi122.stdout: osd op queue = debug_random 2024-03-25T18:15:33.489 INFO:teuthology.orchestra.run.smithi122.stdout: osd op queue cut off = debug_random 2024-03-25T18:15:33.489 INFO:teuthology.orchestra.run.smithi122.stdout: osd shutdown pgref assert = True 2024-03-25T18:15:33.489 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.489 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr] 2024-03-25T18:15:33.489 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-25T18:15:33.489 INFO:teuthology.orchestra.run.smithi122.stdout: debug mgr = 20 2024-03-25T18:15:33.489 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-03-25T18:15:33.490 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-03-25T18:15:33.490 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:15:33.490 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:15:33.490 INFO:teuthology.orchestra.run.smithi122.stdout: mgr/telemetry/nag = false 2024-03-25T18:15:33.490 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.490 INFO:teuthology.orchestra.run.smithi122.stdout:[mon] 2024-03-25T18:15:33.490 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-25T18:15:33.490 INFO:teuthology.orchestra.run.smithi122.stdout: debug mon = 20 2024-03-25T18:15:33.490 INFO:teuthology.orchestra.run.smithi122.stdout: debug paxos = 20 2024-03-25T18:15:33.490 INFO:teuthology.orchestra.run.smithi122.stdout: debug auth = 20 2024-03-25T18:15:33.490 INFO:teuthology.orchestra.run.smithi122.stdout: mon data avail warn = 5 2024-03-25T18:15:33.490 INFO:teuthology.orchestra.run.smithi122.stdout: mon mgr mkfs grace = 240 2024-03-25T18:15:33.490 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:15:33.490 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd reporter subtree level = osd 2024-03-25T18:15:33.490 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd prime pg temp = true 2024-03-25T18:15:33.491 INFO:teuthology.orchestra.run.smithi122.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:15:33.491 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.491 INFO:teuthology.orchestra.run.smithi122.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:15:33.491 INFO:teuthology.orchestra.run.smithi122.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:15:33.491 INFO:teuthology.orchestra.run.smithi122.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:15:33.491 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.491 INFO:teuthology.orchestra.run.smithi122.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:15:33.491 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:15:33.491 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:15:33.491 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.491 INFO:teuthology.orchestra.run.smithi122.stdout: # 1m isn't quite enough 2024-03-25T18:15:33.491 INFO:teuthology.orchestra.run.smithi122.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:15:33.491 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.491 INFO:teuthology.orchestra.run.smithi122.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:15:33.491 INFO:teuthology.orchestra.run.smithi122.stdout: mon min osdmap epochs = 50 2024-03-25T18:15:33.491 INFO:teuthology.orchestra.run.smithi122.stdout: mon osd crush smoke test = False 2024-03-25T18:15:33.492 INFO:teuthology.orchestra.run.smithi122.stdout: mon osdmap full prune interval = 2 2024-03-25T18:15:33.492 INFO:teuthology.orchestra.run.smithi122.stdout: mon osdmap full prune min = 15 2024-03-25T18:15:33.492 INFO:teuthology.orchestra.run.smithi122.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:15:33.492 INFO:teuthology.orchestra.run.smithi122.stdout: mon scrub interval = 300 2024-03-25T18:15:33.492 INFO:teuthology.orchestra.run.smithi122.stdout: mon warn on pool no app = False 2024-03-25T18:15:33.492 INFO:teuthology.orchestra.run.smithi122.stdout: paxos service trim min = 10 2024-03-25T18:15:33.492 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:15:33.492 INFO:teuthology.orchestra.run.smithi122.stdout:[client] 2024-03-25T18:15:33.492 INFO:teuthology.orchestra.run.smithi122.stdout: rgw cache enabled = true 2024-03-25T18:15:33.492 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable ops log = true 2024-03-25T18:15:33.492 INFO:teuthology.orchestra.run.smithi122.stdout: rgw enable usage log = true 2024-03-25T18:15:33.492 INFO:teuthology.orchestra.run.smithi122.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:15:33.493 INFO:teuthology.orchestra.run.smithi122.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:15:33.493 INFO:teuthology.orchestra.run.smithi122.stdout: debug monc = 20 2024-03-25T18:15:33.493 INFO:teuthology.orchestra.run.smithi122.stdout: debug ms = 1 2024-03-25T18:15:33.493 INFO:teuthology.orchestra.run.smithi122.stdout: debug objecter = 20 2024-03-25T18:15:33.493 INFO:teuthology.orchestra.run.smithi122.stdout: debug rados = 20 2024-03-25T18:15:33.493 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.a] 2024-03-25T18:15:33.493 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.c] 2024-03-25T18:15:33.493 INFO:teuthology.orchestra.run.smithi122.stdout:[mon.b] 2024-03-25T18:15:33.493 INFO:teuthology.orchestra.run.smithi204.stdout:[global] 2024-03-25T18:15:33.493 INFO:teuthology.orchestra.run.smithi204.stdout: chdir = "" 2024-03-25T18:15:33.493 INFO:teuthology.orchestra.run.smithi204.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:15:33.494 INFO:teuthology.orchestra.run.smithi204.stdout: auth supported = cephx 2024-03-25T18:15:33.494 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.494 INFO:teuthology.orchestra.run.smithi204.stdout: filestore xattr use omap = true 2024-03-25T18:15:33.494 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.494 INFO:teuthology.orchestra.run.smithi204.stdout: mon clock drift allowed = 1.000 2024-03-25T18:15:33.494 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.494 INFO:teuthology.orchestra.run.smithi204.stdout: osd crush chooseleaf type = 0 2024-03-25T18:15:33.494 INFO:teuthology.orchestra.run.smithi204.stdout: auth debug = true 2024-03-25T18:15:33.494 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.494 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on old message = true 2024-03-25T18:15:33.494 INFO:teuthology.orchestra.run.smithi204.stdout: ms die on bug = true 2024-03-25T18:15:33.494 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.494 INFO:teuthology.orchestra.run.smithi204.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:15:33.494 INFO:teuthology.orchestra.run.smithi204.stdout: mon pg warn max object skew = 0 2024-03-25T18:15:33.494 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.494 INFO:teuthology.orchestra.run.smithi204.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:15:33.495 INFO:teuthology.orchestra.run.smithi204.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:15:33.495 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.495 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default size = 2 2024-03-25T18:15:33.495 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.495 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow primary affinity = true 2024-03-25T18:15:33.495 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd allow pg remap = true 2024-03-25T18:15:33.495 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:15:33.495 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:15:33.495 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on no sortbitwise = false 2024-03-25T18:15:33.495 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:15:33.495 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on too few osds = false 2024-03-25T18:15:33.495 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:15:33.495 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:15:33.495 INFO:teuthology.orchestra.run.smithi204.stdout: mon_allow_pool_size_one = true 2024-03-25T18:15:33.495 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.495 INFO:teuthology.orchestra.run.smithi204.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:15:33.496 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.496 INFO:teuthology.orchestra.run.smithi204.stdout: osd default data pool replay window = 5 2024-03-25T18:15:33.496 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.496 INFO:teuthology.orchestra.run.smithi204.stdout: mon allow pool delete = true 2024-03-25T18:15:33.496 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.496 INFO:teuthology.orchestra.run.smithi204.stdout: mon cluster log file level = debug 2024-03-25T18:15:33.496 INFO:teuthology.orchestra.run.smithi204.stdout: debug asserts on shutdown = true 2024-03-25T18:15:33.496 INFO:teuthology.orchestra.run.smithi204.stdout: mon health detail to clog = false 2024-03-25T18:15:33.496 INFO:teuthology.orchestra.run.smithi204.stdout: mon host = "[v1:172.21.15.122:6789],[v1:172.21.15.122:6790],[v1:172.21.15.204:6789]" 2024-03-25T18:15:33.496 INFO:teuthology.orchestra.run.smithi204.stdout: mon client directed command retry = 5 2024-03-25T18:15:33.496 INFO:teuthology.orchestra.run.smithi204.stdout: mon election default strategy = 3 2024-03-25T18:15:33.496 INFO:teuthology.orchestra.run.smithi204.stdout: ms bind msgr2 = False 2024-03-25T18:15:33.497 INFO:teuthology.orchestra.run.smithi204.stdout: ms inject socket failures = 5000 2024-03-25T18:15:33.497 INFO:teuthology.orchestra.run.smithi204.stdout: ms type = async 2024-03-25T18:15:33.497 INFO:teuthology.orchestra.run.smithi204.stdout: osd heartbeat grace = 80 2024-03-25T18:15:33.497 INFO:teuthology.orchestra.run.smithi204.stdout: fsid = b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:15:33.497 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.497 INFO:teuthology.orchestra.run.smithi204.stdout:[osd] 2024-03-25T18:15:33.497 INFO:teuthology.orchestra.run.smithi204.stdout: osd journal size = 100 2024-03-25T18:15:33.497 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.497 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub load threshold = 5.0 2024-03-25T18:15:33.497 INFO:teuthology.orchestra.run.smithi204.stdout: osd scrub max interval = 600 2024-03-25T18:15:33.497 INFO:teuthology.orchestra.run.smithi204.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:15:33.497 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.497 INFO:teuthology.orchestra.run.smithi204.stdout: osd recover clone overlap = true 2024-03-25T18:15:33.497 INFO:teuthology.orchestra.run.smithi204.stdout: osd recovery max chunk = 1048576 2024-03-25T18:15:33.497 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.497 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug shutdown = true 2024-03-25T18:15:33.497 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug op order = true 2024-03-25T18:15:33.498 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify stray on activate = true 2024-03-25T18:15:33.498 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug trim objects = true 2024-03-25T18:15:33.498 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.498 INFO:teuthology.orchestra.run.smithi204.stdout: osd open classes on start = true 2024-03-25T18:15:33.498 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug pg log writeout = true 2024-03-25T18:15:33.498 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.498 INFO:teuthology.orchestra.run.smithi204.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:15:33.498 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.498 INFO:teuthology.orchestra.run.smithi204.stdout: osd map max advance = 10 2024-03-25T18:15:33.498 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.498 INFO:teuthology.orchestra.run.smithi204.stdout: journal zero on create = true 2024-03-25T18:15:33.498 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.499 INFO:teuthology.orchestra.run.smithi204.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:15:33.499 INFO:teuthology.orchestra.run.smithi204.stdout: filestore apply finisher threads = 3 2024-03-25T18:15:33.499 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.499 INFO:teuthology.orchestra.run.smithi204.stdout: bdev debug aio = true 2024-03-25T18:15:33.499 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug misdirected ops = true 2024-03-25T18:15:33.499 INFO:teuthology.orchestra.run.smithi204.stdout: bdev async discard = True 2024-03-25T18:15:33.499 INFO:teuthology.orchestra.run.smithi204.stdout: bdev enable discard = True 2024-03-25T18:15:33.499 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore allocator = stupid 2024-03-25T18:15:33.499 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore block size = 96636764160 2024-03-25T18:15:33.499 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore fsck on mount = True 2024-03-25T18:15:33.499 INFO:teuthology.orchestra.run.smithi204.stdout: bluestore zero block detection = True 2024-03-25T18:15:33.499 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluefs = 1 2024-03-25T18:15:33.499 INFO:teuthology.orchestra.run.smithi204.stdout: debug bluestore = 1 2024-03-25T18:15:33.499 INFO:teuthology.orchestra.run.smithi204.stdout: debug monc = 20 2024-03-25T18:15:33.499 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-03-25T18:15:33.499 INFO:teuthology.orchestra.run.smithi204.stdout: debug osd = 20 2024-03-25T18:15:33.500 INFO:teuthology.orchestra.run.smithi204.stdout: debug rocksdb = 10 2024-03-25T18:15:33.500 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:15:33.500 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd full ratio = 0.9 2024-03-25T18:15:33.500 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:15:33.500 INFO:teuthology.orchestra.run.smithi204.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:15:33.500 INFO:teuthology.orchestra.run.smithi204.stdout: osd class default list = * 2024-03-25T18:15:33.500 INFO:teuthology.orchestra.run.smithi204.stdout: osd class load list = * 2024-03-25T18:15:33.500 INFO:teuthology.orchestra.run.smithi204.stdout: osd client watch timeout = 120 2024-03-25T18:15:33.500 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify cached snaps = True 2024-03-25T18:15:33.500 INFO:teuthology.orchestra.run.smithi204.stdout: osd debug verify missing on start = True 2024-03-25T18:15:33.500 INFO:teuthology.orchestra.run.smithi204.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:15:33.500 INFO:teuthology.orchestra.run.smithi204.stdout: osd fast shutdown = False 2024-03-25T18:15:33.500 INFO:teuthology.orchestra.run.smithi204.stdout: osd max markdown count = 1000 2024-03-25T18:15:33.500 INFO:teuthology.orchestra.run.smithi204.stdout: osd mclock override recovery settings = True 2024-03-25T18:15:33.500 INFO:teuthology.orchestra.run.smithi204.stdout: osd objectstore = bluestore 2024-03-25T18:15:33.500 INFO:teuthology.orchestra.run.smithi204.stdout: osd op queue = debug_random 2024-03-25T18:15:33.500 INFO:teuthology.orchestra.run.smithi204.stdout: osd op queue cut off = debug_random 2024-03-25T18:15:33.501 INFO:teuthology.orchestra.run.smithi204.stdout: osd shutdown pgref assert = True 2024-03-25T18:15:33.501 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.501 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr] 2024-03-25T18:15:33.501 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-03-25T18:15:33.501 INFO:teuthology.orchestra.run.smithi204.stdout: debug mgr = 20 2024-03-25T18:15:33.501 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-03-25T18:15:33.501 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-03-25T18:15:33.501 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:15:33.501 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:15:33.501 INFO:teuthology.orchestra.run.smithi204.stdout: mgr/telemetry/nag = false 2024-03-25T18:15:33.501 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.501 INFO:teuthology.orchestra.run.smithi204.stdout:[mon] 2024-03-25T18:15:33.501 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-03-25T18:15:33.501 INFO:teuthology.orchestra.run.smithi204.stdout: debug mon = 20 2024-03-25T18:15:33.501 INFO:teuthology.orchestra.run.smithi204.stdout: debug paxos = 20 2024-03-25T18:15:33.501 INFO:teuthology.orchestra.run.smithi204.stdout: debug auth = 20 2024-03-25T18:15:33.501 INFO:teuthology.orchestra.run.smithi204.stdout: mon data avail warn = 5 2024-03-25T18:15:33.502 INFO:teuthology.orchestra.run.smithi204.stdout: mon mgr mkfs grace = 240 2024-03-25T18:15:33.502 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:15:33.502 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd reporter subtree level = osd 2024-03-25T18:15:33.502 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd prime pg temp = true 2024-03-25T18:15:33.502 INFO:teuthology.orchestra.run.smithi204.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:15:33.502 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.502 INFO:teuthology.orchestra.run.smithi204.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:15:33.502 INFO:teuthology.orchestra.run.smithi204.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:15:33.502 INFO:teuthology.orchestra.run.smithi204.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:15:33.502 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.502 INFO:teuthology.orchestra.run.smithi204.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:15:33.502 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:15:33.502 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:15:33.502 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.502 INFO:teuthology.orchestra.run.smithi204.stdout: # 1m isn't quite enough 2024-03-25T18:15:33.502 INFO:teuthology.orchestra.run.smithi204.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout: mon min osdmap epochs = 50 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout: mon osd crush smoke test = False 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout: mon osdmap full prune interval = 2 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout: mon osdmap full prune min = 15 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout: mon scrub interval = 300 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout: mon warn on pool no app = False 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout: paxos service trim min = 10 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout: 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout:[client] 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout: rgw cache enabled = true 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable ops log = true 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout: rgw enable usage log = true 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:15:33.503 INFO:teuthology.orchestra.run.smithi204.stdout: debug monc = 20 2024-03-25T18:15:33.504 INFO:teuthology.orchestra.run.smithi204.stdout: debug ms = 1 2024-03-25T18:15:33.504 INFO:teuthology.orchestra.run.smithi204.stdout: debug objecter = 20 2024-03-25T18:15:33.504 INFO:teuthology.orchestra.run.smithi204.stdout: debug rados = 20 2024-03-25T18:15:33.504 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.a] 2024-03-25T18:15:33.504 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.c] 2024-03-25T18:15:33.504 INFO:teuthology.orchestra.run.smithi204.stdout:[mon.b] 2024-03-25T18:15:33.505 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:15:33.505 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:15:33.560 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:15:33.560 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:15:33.560 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:15:33.575 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:15:33.575 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:15:33.629 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi122.front.sepia.ceph.com 2024-03-25T18:15:33.629 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:15:33.630 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:15:33.630 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:15:33.709 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:15:33.709 DEBUG:teuthology.orchestra.run.smithi122:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:33.763 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi204.front.sepia.ceph.com 2024-03-25T18:15:33.763 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T18:15:33.763 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:15:33.763 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:15:33.803 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T18:15:33.803 DEBUG:teuthology.orchestra.run.smithi204:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:33.859 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:15:33.860 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:15:33.860 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:15:33.915 INFO:teuthology.orchestra.run.smithi122.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:15:33.917 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:15:34.006 INFO:teuthology.orchestra.run.smithi204.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:15:34.009 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:15:34.010 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:15:34.010 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:15:34.052 INFO:teuthology.orchestra.run.smithi122.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:15:34.067 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:15:34.109 INFO:teuthology.orchestra.run.smithi204.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:15:34.124 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:15:34.124 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi122.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi204.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:15:34.124 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:15:34.150 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:34.150 INFO:tasks.ceph:role: osd.0 2024-03-25T18:15:34.151 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com 2024-03-25T18:15:34.151 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:15:34.225 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:15:34.225 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:15:34.225 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:15:34.225 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:15:34.225 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:15:34.225 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:15:34.225 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:15:34.225 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:15:34.225 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:15:34.225 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:15:34.737 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-03-25T18:15:34.738 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-03-25T18:15:34.739 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:15:34.878 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:15:34.912 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:15:34.981 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:34.981 INFO:tasks.ceph:role: osd.1 2024-03-25T18:15:34.981 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com 2024-03-25T18:15:34.981 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:15:35.055 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:15:35.055 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:15:35.055 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:15:35.056 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:15:35.056 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:15:35.056 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:15:35.056 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:15:35.056 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:15:35.056 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:15:35.056 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:15:35.500 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-03-25T18:15:35.502 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-03-25T18:15:35.502 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:15:35.596 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:15:35.627 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:15:35.695 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:35.695 INFO:tasks.ceph:role: osd.2 2024-03-25T18:15:35.695 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com 2024-03-25T18:15:35.696 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:15:35.771 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:15:35.771 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:15:35.771 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:15:35.771 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:15:35.771 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:15:35.771 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:15:35.771 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:15:35.771 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:15:35.771 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:15:35.771 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:15:36.261 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-03-25T18:15:36.263 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-03-25T18:15:36.263 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:15:36.393 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:15:36.426 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:15:36.494 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:36.494 INFO:tasks.ceph:role: osd.3 2024-03-25T18:15:36.495 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi122.front.sepia.ceph.com 2024-03-25T18:15:36.495 DEBUG:teuthology.orchestra.run.smithi122:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:15:36.570 INFO:teuthology.orchestra.run.smithi122.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:15:36.570 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:15:36.570 INFO:teuthology.orchestra.run.smithi122.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:15:36.570 INFO:teuthology.orchestra.run.smithi122.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:15:36.570 INFO:teuthology.orchestra.run.smithi122.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:15:36.570 INFO:teuthology.orchestra.run.smithi122.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:15:36.571 INFO:teuthology.orchestra.run.smithi122.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:15:36.571 INFO:teuthology.orchestra.run.smithi122.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:15:36.571 INFO:teuthology.orchestra.run.smithi122.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:15:36.571 INFO:teuthology.orchestra.run.smithi122.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:15:37.075 INFO:teuthology.orchestra.run.smithi122.stdout:Discarding blocks...Done. 2024-03-25T18:15:37.078 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi122.front.sepia.ceph.com -o noatime 2024-03-25T18:15:37.078 DEBUG:teuthology.orchestra.run.smithi122:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:15:37.205 DEBUG:teuthology.orchestra.run.smithi122:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:15:37.242 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:37.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:37.347+0000 7fbf79393640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:15:37.348 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:37.347+0000 7fbf79393640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:15:37.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:37.347+0000 7fbf79393640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:15:37.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:37.347+0000 7fbf79393640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:15:37.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:37.347+0000 7fbf79393640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:15:37.349 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:37.348+0000 7fbf79393640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:15:40.200 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:15:40.232 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:40.327 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:40.326+0000 7f6147326640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:15:40.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:40.326+0000 7f6147326640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:15:40.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:40.326+0000 7f6147326640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:15:40.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:40.327+0000 7f6147326640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:15:40.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:40.327+0000 7f6147326640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:15:40.328 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:40.327+0000 7f6147326640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:15:43.169 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:15:43.201 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:43.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:43.295+0000 7f5daab6b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:15:43.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:43.295+0000 7f5daab6b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:15:43.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:43.295+0000 7f5daab6b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:15:43.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:43.295+0000 7f5daab6b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:15:43.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:43.295+0000 7f5daab6b640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:15:43.297 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:43.296+0000 7f5daab6b640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:15:46.157 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:15:46.189 DEBUG:teuthology.orchestra.run.smithi122:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:46.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:46.283+0000 7f345b223640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:15:46.284 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:46.283+0000 7f345b223640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:15:46.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:46.283+0000 7f345b223640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:15:46.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:46.283+0000 7f345b223640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:15:46.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:46.283+0000 7f345b223640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:15:46.285 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:15:46.284+0000 7f345b223640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:15:49.123 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:15:49.155 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:15:49.187 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:49.187 INFO:tasks.ceph:role: osd.4 2024-03-25T18:15:49.187 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-25T18:15:49.187 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:15:49.264 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:15:49.265 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:15:49.265 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:15:49.265 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:15:49.265 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:15:49.265 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:15:49.265 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:15:49.265 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:15:49.265 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:15:49.265 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:15:49.774 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-03-25T18:15:49.776 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-03-25T18:15:49.777 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:15:49.929 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:15:49.967 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:15:50.037 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:50.037 INFO:tasks.ceph:role: osd.5 2024-03-25T18:15:50.038 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-25T18:15:50.038 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:15:50.114 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:15:50.114 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:15:50.114 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:15:50.114 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:15:50.114 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:15:50.114 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:15:50.114 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:15:50.114 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:15:50.114 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:15:50.114 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:15:50.559 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-03-25T18:15:50.562 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-03-25T18:15:50.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:15:50.653 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:15:50.684 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:15:50.752 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:50.752 INFO:tasks.ceph:role: osd.6 2024-03-25T18:15:50.752 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-25T18:15:50.752 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:15:50.827 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:15:50.827 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:15:50.827 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:15:50.827 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:15:50.827 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:15:50.827 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:15:50.827 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:15:50.827 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:15:50.828 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:15:50.828 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:15:51.314 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-03-25T18:15:51.315 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-03-25T18:15:51.316 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:15:51.446 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:15:51.480 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:15:51.547 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:15:51.547 INFO:tasks.ceph:role: osd.7 2024-03-25T18:15:51.547 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-25T18:15:51.548 DEBUG:teuthology.orchestra.run.smithi204:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:15:51.621 INFO:teuthology.orchestra.run.smithi204.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:15:51.621 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:15:51.621 INFO:teuthology.orchestra.run.smithi204.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:15:51.621 INFO:teuthology.orchestra.run.smithi204.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:15:51.622 INFO:teuthology.orchestra.run.smithi204.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:15:51.622 INFO:teuthology.orchestra.run.smithi204.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:15:51.622 INFO:teuthology.orchestra.run.smithi204.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:15:51.622 INFO:teuthology.orchestra.run.smithi204.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:15:51.622 INFO:teuthology.orchestra.run.smithi204.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:15:51.622 INFO:teuthology.orchestra.run.smithi204.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:15:52.110 INFO:teuthology.orchestra.run.smithi204.stdout:Discarding blocks...Done. 2024-03-25T18:15:52.111 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi204.front.sepia.ceph.com -o noatime 2024-03-25T18:15:52.112 DEBUG:teuthology.orchestra.run.smithi204:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:15:52.226 DEBUG:teuthology.orchestra.run.smithi204:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:15:52.258 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:52.358 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:52.357+0000 7f8578ca2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:15:52.358 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:52.358+0000 7f8578ca2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:15:52.358 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:52.358+0000 7f8578ca2640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:15:52.358 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:52.358+0000 7f8578ca2640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:15:52.358 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:52.358+0000 7f8578ca2640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:15:52.359 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:52.358+0000 7f8578ca2640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:15:55.198 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:15:55.229 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:55.323 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:55.322+0000 7f8f84cae640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:15:55.323 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:55.322+0000 7f8f84cae640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:15:55.323 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:55.322+0000 7f8f84cae640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:15:55.323 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:55.322+0000 7f8f84cae640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:15:55.323 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:55.322+0000 7f8f84cae640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:15:55.323 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:55.323+0000 7f8f84cae640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:15:58.161 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:15:58.192 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:15:58.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:58.282+0000 7f83fb612640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:15:58.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:58.283+0000 7f83fb612640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:15:58.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:58.283+0000 7f83fb612640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:15:58.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:58.283+0000 7f83fb612640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:15:58.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:58.283+0000 7f83fb612640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:15:58.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:15:58.283+0000 7f83fb612640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:16:01.129 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:16:01.159 DEBUG:teuthology.orchestra.run.smithi204:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:16:01.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:01.256+0000 7fad4d947640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:16:01.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:01.256+0000 7fad4d947640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:16:01.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:01.256+0000 7fad4d947640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:16:01.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:01.256+0000 7fad4d947640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:16:01.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:01.256+0000 7fad4d947640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:16:01.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:01.257+0000 7fad4d947640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:16:04.099 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:16:04.130 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:16:04.130 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:16:04.130 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:16:04.162 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:16:04.162 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:16:04.233 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:16:04.233 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:16:04.302 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:16:04.303 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:16:04.371 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:16:04.371 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:16:04.440 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T18:16:04.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:16:04.471 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T18:16:04.471 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:16:04.542 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T18:16:04.542 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:16:04.612 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T18:16:04.612 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:16:04.681 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T18:16:04.681 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:16:04.749 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:16:04.749 DEBUG:teuthology.orchestra.run.smithi122:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:16:04.766 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T18:16:04.766 DEBUG:teuthology.orchestra.run.smithi204:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:16:04.805 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:16:04.805 DEBUG:teuthology.orchestra.run.smithi122:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:16:04.809 DEBUG:teuthology.orchestra.run.smithi204:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:16:04.849 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr.y] 2024-03-25T18:16:04.849 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBFvwFmSbSONhAAhw/TIpISrg2vYCjCprH+Bw== 2024-03-25T18:16:04.849 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.0] 2024-03-25T18:16:04.849 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBJvwFmVF3EFBAAUo6oE2KAf7PudXv94E1kqQ== 2024-03-25T18:16:04.849 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.1] 2024-03-25T18:16:04.849 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBMvwFmW2CIExAAQ1OmjQF68e1H23b5WKYZCQ== 2024-03-25T18:16:04.849 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.2] 2024-03-25T18:16:04.850 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBPvwFmjuasERAAEZ3umh8zcCtoh1BucVoYNw== 2024-03-25T18:16:04.850 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.3] 2024-03-25T18:16:04.850 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBSvwFmNp/0EBAAWCRLGhuTh7BJvntGkbd0pg== 2024-03-25T18:16:04.850 INFO:teuthology.orchestra.run.smithi122.stdout:[mgr.x] 2024-03-25T18:16:04.850 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBGvwFmOQxxABAATRHeut4Un12VDdQURmzUWw== 2024-03-25T18:16:04.850 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.4] 2024-03-25T18:16:04.850 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBYvwFmNr1iFRAArNSVj/exLz/L0+hqJRksLQ== 2024-03-25T18:16:04.850 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.5] 2024-03-25T18:16:04.850 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBbvwFmBZRFExAAwK4NxatQSFMoXFkRzDBQCQ== 2024-03-25T18:16:04.850 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.6] 2024-03-25T18:16:04.850 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBevwFmZ/TnEBAAUVim2TXcs+8LcH6DOTJXOA== 2024-03-25T18:16:04.850 INFO:teuthology.orchestra.run.smithi122.stdout:[osd.7] 2024-03-25T18:16:04.850 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBhvwFmGfZVDxAAEpeF78b6xUHiUN6BSkKJ0g== 2024-03-25T18:16:04.850 INFO:teuthology.orchestra.run.smithi122.stdout:[client.0] 2024-03-25T18:16:04.850 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBGvwFmZr0dAxAALzyGteSPP4tx126gTpxeyQ== 2024-03-25T18:16:04.850 INFO:teuthology.orchestra.run.smithi122.stdout:[client.1] 2024-03-25T18:16:04.851 INFO:teuthology.orchestra.run.smithi122.stdout: key = AQBGvwFmdB6KBhAAYkQFNC6xCzU1+yPp3a8ung== 2024-03-25T18:16:04.872 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr.y] 2024-03-25T18:16:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBFvwFmSbSONhAAhw/TIpISrg2vYCjCprH+Bw== 2024-03-25T18:16:04.872 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.0] 2024-03-25T18:16:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBJvwFmVF3EFBAAUo6oE2KAf7PudXv94E1kqQ== 2024-03-25T18:16:04.872 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.1] 2024-03-25T18:16:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBMvwFmW2CIExAAQ1OmjQF68e1H23b5WKYZCQ== 2024-03-25T18:16:04.872 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.2] 2024-03-25T18:16:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBPvwFmjuasERAAEZ3umh8zcCtoh1BucVoYNw== 2024-03-25T18:16:04.872 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.3] 2024-03-25T18:16:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBSvwFmNp/0EBAAWCRLGhuTh7BJvntGkbd0pg== 2024-03-25T18:16:04.872 INFO:teuthology.orchestra.run.smithi204.stdout:[mgr.x] 2024-03-25T18:16:04.872 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBGvwFmOQxxABAATRHeut4Un12VDdQURmzUWw== 2024-03-25T18:16:04.872 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.4] 2024-03-25T18:16:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBYvwFmNr1iFRAArNSVj/exLz/L0+hqJRksLQ== 2024-03-25T18:16:04.873 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.5] 2024-03-25T18:16:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBbvwFmBZRFExAAwK4NxatQSFMoXFkRzDBQCQ== 2024-03-25T18:16:04.873 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.6] 2024-03-25T18:16:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBevwFmZ/TnEBAAUVim2TXcs+8LcH6DOTJXOA== 2024-03-25T18:16:04.873 INFO:teuthology.orchestra.run.smithi204.stdout:[osd.7] 2024-03-25T18:16:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBhvwFmGfZVDxAAEpeF78b6xUHiUN6BSkKJ0g== 2024-03-25T18:16:04.873 INFO:teuthology.orchestra.run.smithi204.stdout:[client.0] 2024-03-25T18:16:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBGvwFmZr0dAxAALzyGteSPP4tx126gTpxeyQ== 2024-03-25T18:16:04.873 INFO:teuthology.orchestra.run.smithi204.stdout:[client.1] 2024-03-25T18:16:04.873 INFO:teuthology.orchestra.run.smithi204.stdout: key = AQBGvwFmdB6KBhAAYkQFNC6xCzU1+yPp3a8ung== 2024-03-25T18:16:04.874 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:16:04.894 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:16:04.967 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:16:04.969 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:16:05.021 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:16:05.023 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:16:05.075 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:16:05.077 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:16:05.129 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:16:05.131 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:16:05.182 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:16:05.185 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:16:05.236 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:16:05.238 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:16:05.289 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:16:05.291 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:16:05.343 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:16:05.345 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:16:05.396 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:16:05.399 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:16:05.450 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:16:05.452 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:16:05.506 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:16:05.509 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:16:05.563 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:16:05.563 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:16:05.589 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:16:05.686 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:16:05.712 DEBUG:teuthology.orchestra.run.smithi122:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:16:05.779 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:16:05.874 DEBUG:teuthology.orchestra.run.smithi122:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:16:05.901 DEBUG:teuthology.orchestra.run.smithi204:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:16:05.929 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:16:06.026 DEBUG:teuthology.orchestra.run.smithi204:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:16:06.052 DEBUG:teuthology.orchestra.run.smithi122:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:16:06.055 DEBUG:teuthology.orchestra.run.smithi204:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:16:06.107 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:16:06.107 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:16:06.107 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:16:06.107 DEBUG:teuthology.orchestra.run.smithi122:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:16:06.110 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:16:06.110 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:16:06.110 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:16:06.110 DEBUG:teuthology.orchestra.run.smithi122:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:16:06.113 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:16:06.113 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:16:06.113 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:16:06.113 DEBUG:teuthology.orchestra.run.smithi204:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:16:06.149 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:16:06.149 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:16:06.150 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:16:06.150 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:16:06.152 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:16:06.152 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:16:06.152 DEBUG:teuthology.orchestra.run.smithi204:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:16:06.154 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:16:06.154 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:16:06.155 DEBUG:teuthology.orchestra.run.smithi122:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:16:06.255 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.254+0000 7f75daff4640 10 monclient: get_monmap_and_config 2024-03-25T18:16:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.254+0000 7f75daff4640 10 monclient: build_initial_monmap 2024-03-25T18:16:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.254+0000 7f75daff4640 10 monclient: monmap: 2024-03-25T18:16:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 0 2024-03-25T18:16:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-03-25T18:16:06.256023+0000 2024-03-25T18:16:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-03-25T18:16:06.256023+0000 2024-03-25T18:16:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:16:06.256 INFO:teuthology.orchestra.run.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:16:06.256 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-25T18:16:06.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.255+0000 7f75daff4640 1 Processor -- start 2024-03-25T18:16:06.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.255+0000 7f75daff4640 1 -- start start 2024-03-25T18:16:06.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.255+0000 7f75daff4640 10 monclient: init 2024-03-25T18:16:06.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.256+0000 7f75daff4640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:06.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.256+0000 7f75daff4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:06.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.256+0000 7f75daff4640 10 monclient(hunting): picked mon.noname-b con 0x7f75d40b4700 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:06.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.256+0000 7f75daff4640 10 monclient(hunting): picked mon.noname-c con 0x7f75d4142a20 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:06.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.256+0000 7f75daff4640 10 monclient(hunting): picked mon.noname-a con 0x7f75d40a96b0 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:06.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.256+0000 7f75daff4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d40a9d60 con 0x7f75d40a96b0 2024-03-25T18:16:06.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.256+0000 7f75daff4640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d413ff10 con 0x7f75d4142a20 2024-03-25T18:16:06.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.256+0000 7f75daff4640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d40aa1f0 con 0x7f75d40b4700 2024-03-25T18:16:06.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.256+0000 7f75d8d90640 1 -- >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75d4142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6790/0 2024-03-25T18:16:06.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.256+0000 7f75daff4640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:06.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.256+0000 7f75d3fff640 1 -- >> v1:172.21.15.204:6789/0 conn(0x7f75d4142a20 legacy=0x7f75d413f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.204:6789/0 2024-03-25T18:16:06.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.256+0000 7f75daff4640 10 monclient(hunting): authenticate will time out at 7289.522493s 2024-03-25T18:16:06.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.256+0000 7f75d9591640 1 -- >> v1:172.21.15.122:6789/0 conn(0x7f75d40a96b0 legacy=0x7f75d413f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-03-25T18:16:06.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.457+0000 7f75d8d90640 1 -- >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75d4142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6790/0 2024-03-25T18:16:06.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.457+0000 7f75d9591640 1 -- >> v1:172.21.15.122:6789/0 conn(0x7f75d40a96b0 legacy=0x7f75d413f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-03-25T18:16:06.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.457+0000 7f75d3fff640 1 -- >> v1:172.21.15.204:6789/0 conn(0x7f75d4142a20 legacy=0x7f75d413f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.204:6789/0 2024-03-25T18:16:06.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.858+0000 7f75d8d90640 1 -- >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75d4142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6790/0 2024-03-25T18:16:06.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.858+0000 7f75d9591640 1 -- >> v1:172.21.15.122:6789/0 conn(0x7f75d40a96b0 legacy=0x7f75d413f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-03-25T18:16:06.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:06.858+0000 7f75d3fff640 1 -- >> v1:172.21.15.204:6789/0 conn(0x7f75d4142a20 legacy=0x7f75d413f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.204:6789/0 2024-03-25T18:16:07.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:07.659+0000 7f75d8d90640 1 -- >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75d4142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6790/0 2024-03-25T18:16:07.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:07.659+0000 7f75d9591640 1 -- >> v1:172.21.15.122:6789/0 conn(0x7f75d40a96b0 legacy=0x7f75d413f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-03-25T18:16:07.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:07.659+0000 7f75d3fff640 1 -- >> v1:172.21.15.204:6789/0 conn(0x7f75d4142a20 legacy=0x7f75d413f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.204:6789/0 2024-03-25T18:16:09.257 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d27fc640 10 monclient(hunting): tick 2024-03-25T18:16:09.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d27fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:16:09.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d27fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:16:09.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d27fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:16:09.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d27fc640 1 -- >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75d4142310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:09.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d27fc640 1 -- >> v1:172.21.15.204:6789/0 conn(0x7f75d4142a20 legacy=0x7f75d413f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:09.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d27fc640 1 -- >> v1:172.21.15.122:6789/0 conn(0x7f75d40a96b0 legacy=0x7f75d413f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:09.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d27fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:09.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-c con 0x7f75d4142a20 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:09.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-a con 0x7f75d40a96b0 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:09.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-b con 0x7f75d40b4700 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:09.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d27fc640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d40b4ae0 con 0x7f75d40a96b0 2024-03-25T18:16:09.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d27fc640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d40a9d60 con 0x7f75d4142a20 2024-03-25T18:16:09.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d27fc640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d413ff10 con 0x7f75d40b4700 2024-03-25T18:16:09.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d27fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:09.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d3fff640 1 -- >> v1:172.21.15.122:6789/0 conn(0x7f75d40a96b0 legacy=0x7f75b8008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-03-25T18:16:09.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d9591640 1 -- >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75b800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6790/0 2024-03-25T18:16:09.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.256+0000 7f75d8d90640 1 -- >> v1:172.21.15.204:6789/0 conn(0x7f75d4142a20 legacy=0x7f75b80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.204:6789/0 2024-03-25T18:16:09.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.457+0000 7f75d3fff640 1 -- >> v1:172.21.15.122:6789/0 conn(0x7f75d40a96b0 legacy=0x7f75b8008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-03-25T18:16:09.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.457+0000 7f75d9591640 1 -- >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75b800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6790/0 2024-03-25T18:16:09.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.457+0000 7f75d8d90640 1 -- >> v1:172.21.15.204:6789/0 conn(0x7f75d4142a20 legacy=0x7f75b80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.204:6789/0 2024-03-25T18:16:09.858 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.858+0000 7f75d3fff640 1 -- >> v1:172.21.15.122:6789/0 conn(0x7f75d40a96b0 legacy=0x7f75b8008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-03-25T18:16:09.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.858+0000 7f75d9591640 1 -- >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75b800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6790/0 2024-03-25T18:16:09.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:09.858+0000 7f75d8d90640 1 -- >> v1:172.21.15.204:6789/0 conn(0x7f75d4142a20 legacy=0x7f75b80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.204:6789/0 2024-03-25T18:16:10.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:10.659+0000 7f75d3fff640 1 -- >> v1:172.21.15.122:6789/0 conn(0x7f75d40a96b0 legacy=0x7f75b8008ed0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-03-25T18:16:10.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:10.659+0000 7f75d9591640 1 -- >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75b800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6790/0 2024-03-25T18:16:10.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:10.659+0000 7f75d8d90640 1 -- >> v1:172.21.15.204:6789/0 conn(0x7f75d4142a20 legacy=0x7f75b80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.204:6789/0 2024-03-25T18:16:12.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d27fc640 10 monclient(hunting): tick 2024-03-25T18:16:12.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d27fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:16:12.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d27fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:16:12.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d27fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:16:12.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d27fc640 1 -- >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75b800c790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:12.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d27fc640 1 -- >> v1:172.21.15.204:6789/0 conn(0x7f75d4142a20 legacy=0x7f75b80056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:12.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d27fc640 1 -- >> v1:172.21.15.122:6789/0 conn(0x7f75d40a96b0 legacy=0x7f75b8008ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:12.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d27fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:12.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-b con 0x7f75d40b4700 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:12.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-a con 0x7f75d40a96b0 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:12.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-c con 0x7f75d4142a20 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:12.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d27fc640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d40aa1f0 con 0x7f75d40a96b0 2024-03-25T18:16:12.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d27fc640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d40b4ae0 con 0x7f75d4142a20 2024-03-25T18:16:12.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d27fc640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d40a9d60 con 0x7f75d40b4700 2024-03-25T18:16:12.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d27fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:12.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d3fff640 1 -- >> v1:172.21.15.122:6789/0 conn(0x7f75d40a96b0 legacy=0x7f75b800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-03-25T18:16:12.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d8d90640 1 -- >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75b800cec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6790/0 2024-03-25T18:16:12.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.256+0000 7f75d9591640 1 -- >> v1:172.21.15.204:6789/0 conn(0x7f75d4142a20 legacy=0x7f75b80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.204:6789/0 2024-03-25T18:16:12.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.457+0000 7f75d3fff640 1 -- >> v1:172.21.15.122:6789/0 conn(0x7f75d40a96b0 legacy=0x7f75b800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-03-25T18:16:12.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.457+0000 7f75d8d90640 1 -- >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75b800cec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6790/0 2024-03-25T18:16:12.458 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.457+0000 7f75d9591640 1 -- >> v1:172.21.15.204:6789/0 conn(0x7f75d4142a20 legacy=0x7f75b80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.204:6789/0 2024-03-25T18:16:12.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.858+0000 7f75d8d90640 1 -- >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75b800cec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6790/0 2024-03-25T18:16:12.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.858+0000 7f75d3fff640 1 -- >> v1:172.21.15.122:6789/0 conn(0x7f75d40a96b0 legacy=0x7f75b800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-03-25T18:16:12.859 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:12.858+0000 7f75d9591640 1 -- >> v1:172.21.15.204:6789/0 conn(0x7f75d4142a20 legacy=0x7f75b80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.204:6789/0 2024-03-25T18:16:13.152 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:00:06.898 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:13.326 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:00:07.069 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:13.327 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:00:07.070 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:13.461 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:00:07.207 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:13.635 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:00:07.378 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:13.635 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:00:07.378 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:13.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:13.659+0000 7f75d3fff640 1 -- >> v1:172.21.15.122:6789/0 conn(0x7f75d40a96b0 legacy=0x7f75b800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6789/0 2024-03-25T18:16:13.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:13.659+0000 7f75d8d90640 1 -- >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75b800cec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.122:6790/0 2024-03-25T18:16:13.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:13.659+0000 7f75d9591640 1 -- >> v1:172.21.15.204:6789/0 conn(0x7f75d4142a20 legacy=0x7f75b80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.204:6789/0 2024-03-25T18:16:15.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:15.256+0000 7f75d27fc640 10 monclient(hunting): tick 2024-03-25T18:16:15.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:15.256+0000 7f75d27fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:16:15.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:15.256+0000 7f75d27fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:16:15.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:15.256+0000 7f75d27fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:16:15.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:15.256+0000 7f75d27fc640 1 -- >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75b800cec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:15.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:15.256+0000 7f75d27fc640 1 -- >> v1:172.21.15.204:6789/0 conn(0x7f75d4142a20 legacy=0x7f75b80056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:15.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:15.256+0000 7f75d27fc640 1 -- >> v1:172.21.15.122:6789/0 conn(0x7f75d40a96b0 legacy=0x7f75b800c790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:16:15.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:15.256+0000 7f75d27fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:15.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:15.256+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-c con 0x7f75d4142a20 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:15.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:15.256+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-a con 0x7f75d40a96b0 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:15.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:15.256+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-b con 0x7f75d40b4700 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:15.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:15.256+0000 7f75d27fc640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d413ff10 con 0x7f75d40a96b0 2024-03-25T18:16:15.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:15.256+0000 7f75d27fc640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d40aa1f0 con 0x7f75d4142a20 2024-03-25T18:16:15.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:15.256+0000 7f75d27fc640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d40b4ae0 con 0x7f75d40b4700 2024-03-25T18:16:15.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:15.256+0000 7f75d27fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:15.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:15.265+0000 7f75d3fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f75d40a96b0 0x7f75b800c790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:55126/0 (socket says 172.21.15.122:55126) 2024-03-25T18:16:15.266 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:15.265+0000 7f75d3fff640 1 -- 172.21.15.122:0/1034250796 learned_addr learned my addr 172.21.15.122:0/1034250796 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:16:18.258 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:18.257+0000 7f75d27fc640 10 monclient(hunting): tick 2024-03-25T18:16:18.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:18.257+0000 7f75d27fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:16:18.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:18.257+0000 7f75d27fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:16:18.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:18.257+0000 7f75d27fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:16:18.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:18.257+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75b800cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:18.257+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 >> v1:172.21.15.204:6789/0 conn(0x7f75d4142a20 legacy=0x7f75b8008ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:18.257+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 >> v1:172.21.15.122:6789/0 conn(0x7f75d40a96b0 legacy=0x7f75b800c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:18.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:18.257+0000 7f75d27fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:18.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:18.257+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-c con 0x7f75b80057a0 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:18.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:18.257+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-a con 0x7f75b8008ec0 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:18.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:18.257+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-b con 0x7f75d40b4700 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:18.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:18.257+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d40a9d60 con 0x7f75b8008ec0 2024-03-25T18:16:18.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:18.257+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b800a2b0 con 0x7f75b80057a0 2024-03-25T18:16:18.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:18.257+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b800a440 con 0x7f75d40b4700 2024-03-25T18:16:18.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:18.257+0000 7f75d27fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:21.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:21.257+0000 7f75d27fc640 10 monclient(hunting): tick 2024-03-25T18:16:21.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:21.257+0000 7f75d27fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:16:21.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:21.257+0000 7f75d27fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:16:21.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:21.257+0000 7f75d27fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:16:21.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:21.257+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75b800cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:21.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:21.257+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 >> v1:172.21.15.204:6789/0 conn(0x7f75b80057a0 legacy=0x7f75b800d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:21.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:21.257+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 >> v1:172.21.15.122:6789/0 conn(0x7f75b8008ec0 legacy=0x7f75b8005e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:21.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:21.257+0000 7f75d27fc640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:21.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:21.257+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-a con 0x7f75b800c970 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:21.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:21.257+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-c con 0x7f75b80057a0 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:21.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:21.257+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-b con 0x7f75d40b4700 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:21.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:21.257+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b800c7a0 con 0x7f75b800c970 2024-03-25T18:16:21.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:21.257+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b8005de0 con 0x7f75b80057a0 2024-03-25T18:16:21.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:21.257+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b8005f70 con 0x7f75d40b4700 2024-03-25T18:16:21.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:21.257+0000 7f75d27fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:24.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:24.258+0000 7f75d27fc640 10 monclient(hunting): tick 2024-03-25T18:16:24.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:24.258+0000 7f75d27fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:16:24.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:24.258+0000 7f75d27fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:16:24.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:24.258+0000 7f75d27fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:16:24.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:24.258+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75b800cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:24.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:24.258+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 >> v1:172.21.15.204:6789/0 conn(0x7f75b80057a0 legacy=0x7f75b800d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:24.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:24.258+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 >> v1:172.21.15.122:6789/0 conn(0x7f75b800c970 legacy=0x7f75b8006510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:24.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:24.258+0000 7f75d27fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:24.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:24.258+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-c con 0x7f75b8006e40 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:24.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:24.258+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-a con 0x7f75b800c970 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:24.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:24.258+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-b con 0x7f75d40b4700 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:24.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:24.258+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b8006150 con 0x7f75b800c970 2024-03-25T18:16:24.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:24.258+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b8008a20 con 0x7f75b8006e40 2024-03-25T18:16:24.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:24.258+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b8008c50 con 0x7f75d40b4700 2024-03-25T18:16:24.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:24.258+0000 7f75d27fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:27.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:27.258+0000 7f75d27fc640 10 monclient(hunting): tick 2024-03-25T18:16:27.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:27.258+0000 7f75d27fc640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:16:27.259 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:27.258+0000 7f75d27fc640 1 monclient(hunting): continuing hunt 2024-03-25T18:16:27.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:27.258+0000 7f75d27fc640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:16:27.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:27.258+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 >> v1:172.21.15.122:6790/0 conn(0x7f75d40b4700 legacy=0x7f75b800cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:27.258+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 >> v1:172.21.15.204:6789/0 conn(0x7f75b8006e40 legacy=0x7f75b8007260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:27.258+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 >> v1:172.21.15.122:6789/0 conn(0x7f75b800c970 legacy=0x7f75b800d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:27.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:27.258+0000 7f75d27fc640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:27.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:27.258+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-b con 0x7f75b80065d0 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:27.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:27.258+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-a con 0x7f75b800c970 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:27.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:27.258+0000 7f75d27fc640 10 monclient(hunting): picked mon.noname-c con 0x7f75b8006e40 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:27.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:27.258+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b8006400 con 0x7f75b800c970 2024-03-25T18:16:27.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:27.258+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b8008270 con 0x7f75b8006e40 2024-03-25T18:16:27.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:27.258+0000 7f75d27fc640 1 -- 172.21.15.122:0/1034250796 --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75b8008400 con 0x7f75b80065d0 2024-03-25T18:16:27.260 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:27.258+0000 7f75d27fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:29.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.497+0000 7f75d37fe640 1 -- 172.21.15.122:0/1034250796 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499227958 0 0) 0x7f75b8006400 con 0x7f75b800c970 2024-03-25T18:16:29.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.497+0000 7f75d37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:29.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.497+0000 7f75d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:29.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.497+0000 7f75d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:29.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.497+0000 7f75d37fe640 10 monclient(hunting): my global_id is 4101 2024-03-25T18:16:29.499 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.497+0000 7f75d37fe640 1 -- 172.21.15.122:0/1034250796 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75c4003520 con 0x7f75b800c970 2024-03-25T18:16:29.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.530+0000 7f75d37fe640 1 -- 172.21.15.122:0/1034250796 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154467054 0 0) 0x7f75b8008400 con 0x7f75b80065d0 2024-03-25T18:16:29.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.530+0000 7f75d37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:29.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.530+0000 7f75d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:29.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.530+0000 7f75d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:29.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.530+0000 7f75d37fe640 10 monclient(hunting): my global_id is 4106 2024-03-25T18:16:29.531 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.530+0000 7f75d37fe640 1 -- 172.21.15.122:0/1034250796 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b8006400 con 0x7f75b80065d0 2024-03-25T18:16:29.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.536+0000 7f75d37fe640 1 -- 172.21.15.122:0/1034250796 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294479142 0 0) 0x7f75b8008270 con 0x7f75b8006e40 2024-03-25T18:16:29.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.536+0000 7f75d37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:29.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.536+0000 7f75d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:29.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.536+0000 7f75d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:29.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.536+0000 7f75d37fe640 10 monclient(hunting): my global_id is 4105 2024-03-25T18:16:29.537 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.536+0000 7f75d37fe640 1 -- 172.21.15.122:0/1034250796 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b8008400 con 0x7f75b8006e40 2024-03-25T18:16:29.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75d37fe640 1 -- 172.21.15.122:0/1034250796 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 47817630 0 0) 0x7f75c4003520 con 0x7f75b800c970 2024-03-25T18:16:29.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75d37fe640 1 -- 172.21.15.122:0/1034250796 >> v1:172.21.15.122:6790/0 conn(0x7f75b80065d0 legacy=0x7f75b8011090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:29.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75d37fe640 1 -- 172.21.15.122:0/1034250796 >> v1:172.21.15.204:6789/0 conn(0x7f75b8006e40 legacy=0x7f75b800cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:29.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75d37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:29.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75d37fe640 1 monclient: found mon.noname-a 2024-03-25T18:16:29.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:29.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75d37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:16:29.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75d37fe640 1 -- 172.21.15.122:0/1034250796 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75b8008590 con 0x7f75b800c970 2024-03-25T18:16:29.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75d37fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:29.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75d37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:29.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:29.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75d37fe640 1 -- 172.21.15.122:0/1034250796 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f75c0004100 con 0x7f75b800c970 2024-03-25T18:16:29.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75daff4640 5 monclient: authenticate success, global_id 4101 2024-03-25T18:16:29.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75daff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:29.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:29.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75d37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:29.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75d37fe640 10 monclient: dump: 2024-03-25T18:16:29.555 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-03-25T18:16:29.555 INFO:teuthology.orchestra.run.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:29.555 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:29.555 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:29.555 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:29.555 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:29.555 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:29.555 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:29.555 INFO:teuthology.orchestra.run.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:29.555 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-25T18:16:29.555 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.552+0000 7f75daff4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:29.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.639+0000 7f75d37fe640 1 -- 172.21.15.122:0/1034250796 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75c0004260 con 0x7f75b800c970 2024-03-25T18:16:29.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.639+0000 7f75d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:29.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.639+0000 7f75d37fe640 1 -- 172.21.15.122:0/1034250796 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f75c0004670 con 0x7f75b800c970 2024-03-25T18:16:29.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.639+0000 7f75d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:29.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.639+0000 7f75d37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:29.640 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.639+0000 7f75d37fe640 10 monclient: dump: 2024-03-25T18:16:29.640 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-03-25T18:16:29.640 INFO:teuthology.orchestra.run.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:29.640 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:29.641 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:29.641 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:29.641 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:29.641 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:29.641 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:29.641 INFO:teuthology.orchestra.run.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:29.641 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-25T18:16:29.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.639+0000 7f75daff4640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:29.641 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.639+0000 7f75daff4640 4 set_mon_vals no callback set 2024-03-25T18:16:29.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.640+0000 7f75daff4640 10 monclient: shutdown 2024-03-25T18:16:29.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.640+0000 7f75daff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:29.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.640+0000 7f75daff4640 1 -- 172.21.15.122:0/1034250796 >> v1:172.21.15.122:6789/0 conn(0x7f75b800c970 legacy=0x7f75b800d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:29.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.640+0000 7f75daff4640 1 -- 172.21.15.122:0/1034250796 shutdown_connections 2024-03-25T18:16:29.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.640+0000 7f75daff4640 1 -- 172.21.15.122:0/1034250796 >> 172.21.15.122:0/1034250796 conn(0x7f75d40668e0 msgr2=0x7f75d4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:29.643 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.640+0000 7f75daff4640 1 -- 172.21.15.122:0/1034250796 shutdown_connections 2024-03-25T18:16:29.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.640+0000 7f75daff4640 1 -- 172.21.15.122:0/1034250796 wait complete. 2024-03-25T18:16:29.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.640+0000 7f75daff4640 10 monclient: build_initial_monmap 2024-03-25T18:16:29.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.640+0000 7f75daff4640 10 monclient: monmap: 2024-03-25T18:16:29.644 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 0 2024-03-25T18:16:29.644 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:29.644 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 0.000000 2024-03-25T18:16:29.644 INFO:teuthology.orchestra.run.smithi122.stderr:created 0.000000 2024-03-25T18:16:29.644 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:29.644 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-03-25T18:16:29.644 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:29.644 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:16:29.644 INFO:teuthology.orchestra.run.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:16:29.644 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-25T18:16:29.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.641+0000 7f75daff4640 1 librados: starting msgr at 2024-03-25T18:16:29.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.641+0000 7f75daff4640 1 librados: starting objecter 2024-03-25T18:16:29.644 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.641+0000 7f75daff4640 1 Processor -- start 2024-03-25T18:16:29.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.641+0000 7f75daff4640 1 -- start start 2024-03-25T18:16:29.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.641+0000 7f75daff4640 1 librados: setting wanted keys 2024-03-25T18:16:29.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.641+0000 7f75daff4640 1 librados: calling monclient init 2024-03-25T18:16:29.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.641+0000 7f75daff4640 10 monclient: init 2024-03-25T18:16:29.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.641+0000 7f75daff4640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:29.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.641+0000 7f75daff4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:29.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.641+0000 7f75daff4640 10 monclient(hunting): picked mon.noname-b con 0x7f75b80065d0 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:29.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.641+0000 7f75daff4640 10 monclient(hunting): picked mon.noname-c con 0x7f75b8006e40 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:29.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.641+0000 7f75daff4640 10 monclient(hunting): picked mon.noname-a con 0x7f75b800c970 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:29.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.641+0000 7f75daff4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d41c9f60 con 0x7f75b800c970 2024-03-25T18:16:29.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.641+0000 7f75daff4640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d41cb090 con 0x7f75b80065d0 2024-03-25T18:16:29.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.641+0000 7f75daff4640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75d413bc90 con 0x7f75b8006e40 2024-03-25T18:16:29.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.641+0000 7f75daff4640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:29.645 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.641+0000 7f75daff4640 10 monclient(hunting): authenticate will time out at 7312.907899s 2024-03-25T18:16:29.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.656+0000 7f75d3fff640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f75b8006e40 0x7f75d41c8a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:48492/0 (socket says 172.21.15.122:48492) 2024-03-25T18:16:29.657 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.656+0000 7f75d3fff640 1 -- 172.21.15.122:0/2973724630 learned_addr learned my addr 172.21.15.122:0/2973724630 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:16:29.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.657+0000 7f75d17fa640 10 client.?.objecter ms_handle_connect 0x7f75b8006e40 2024-03-25T18:16:29.658 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.657+0000 7f75d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:29.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.659+0000 7f75d17fa640 10 client.?.objecter ms_handle_connect 0x7f75b80065d0 2024-03-25T18:16:29.660 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.659+0000 7f75d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:29.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.667+0000 7f75d17fa640 10 client.?.objecter ms_handle_connect 0x7f75b800c970 2024-03-25T18:16:29.667 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.667+0000 7f75d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:29.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.728+0000 7f75d17fa640 1 -- 172.21.15.122:0/2973724630 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437526343 0 0) 0x7f75d41c9f60 con 0x7f75b800c970 2024-03-25T18:16:29.729 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.729+0000 7f75d17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:29.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.729+0000 7f75d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:29.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.729+0000 7f75d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:29.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.729+0000 7f75d17fa640 10 monclient(hunting): my global_id is 4113 2024-03-25T18:16:29.730 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.729+0000 7f75d17fa640 1 -- 172.21.15.122:0/2973724630 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75b8008720 con 0x7f75b800c970 2024-03-25T18:16:29.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.776+0000 7f75d17fa640 1 -- 172.21.15.122:0/2973724630 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1754899398 0 0) 0x7f75b8008720 con 0x7f75b800c970 2024-03-25T18:16:29.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.776+0000 7f75d17fa640 1 -- 172.21.15.122:0/2973724630 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75d41c9f60 con 0x7f75b800c970 2024-03-25T18:16:29.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.776+0000 7f75d17fa640 1 -- 172.21.15.122:0/2973724630 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f75cc0044d0 con 0x7f75b800c970 2024-03-25T18:16:29.777 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.776+0000 7f75d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:29.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.776+0000 7f75d17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:29.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.776+0000 7f75d17fa640 10 monclient(hunting): dump: 2024-03-25T18:16:29.778 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-03-25T18:16:29.778 INFO:teuthology.orchestra.run.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:29.778 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:29.778 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:29.778 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:29.778 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:29.778 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:29.778 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:29.778 INFO:teuthology.orchestra.run.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:29.778 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-25T18:16:29.778 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.776+0000 7f75d17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:29.781 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75d17fa640 1 -- 172.21.15.122:0/2973724630 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2943872693 0 0) 0x7f75d41c9f60 con 0x7f75b800c970 2024-03-25T18:16:29.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75d17fa640 1 -- 172.21.15.122:0/2973724630 >> v1:172.21.15.122:6790/0 conn(0x7f75b8006e40 legacy=0x7f75d41c8a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:29.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75d17fa640 1 -- 172.21.15.122:0/2973724630 >> v1:172.21.15.204:6789/0 conn(0x7f75b80065d0 legacy=0x7f75d40af580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:29.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75d17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:16:29.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75d17fa640 1 monclient: found mon.a 2024-03-25T18:16:29.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75d17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:29.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75d17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:29.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75d17fa640 1 -- 172.21.15.122:0/2973724630 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75d413be20 con 0x7f75b800c970 2024-03-25T18:16:29.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75d17fa640 10 monclient: _finish_auth 0 2024-03-25T18:16:29.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75d17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:16:29.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:29.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75daff4640 5 monclient: authenticate success, global_id 4113 2024-03-25T18:16:29.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75daff4640 10 monclient: _renew_subs 2024-03-25T18:16:29.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75daff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:29.782 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75daff4640 1 -- 172.21.15.122:0/2973724630 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75d41dad90 con 0x7f75b800c970 2024-03-25T18:16:29.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75daff4640 10 client.4113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:29.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75daff4640 10 monclient: _renew_subs 2024-03-25T18:16:29.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75daff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:29.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75daff4640 1 -- 172.21.15.122:0/2973724630 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75d41c9850 con 0x7f75b800c970 2024-03-25T18:16:29.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.780+0000 7f75daff4640 1 librados: init done 2024-03-25T18:16:29.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.781+0000 7f75daff4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:29.783 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.781+0000 7f75daff4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:29.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.781+0000 7f75daff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:29.784 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.781+0000 7f75daff4640 1 -- 172.21.15.122:0/2973724630 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75d41c9850 con 0x7f75b800c970 2024-03-25T18:16:29.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.984+0000 7f75d17fa640 1 -- 172.21.15.122:0/2973724630 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75cc0048e0 con 0x7f75b800c970 2024-03-25T18:16:29.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.984+0000 7f75d17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:29.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.984+0000 7f75d17fa640 1 -- 172.21.15.122:0/2973724630 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f75cc004d30 con 0x7f75b800c970 2024-03-25T18:16:29.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.984+0000 7f75d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:29.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.984+0000 7f75d17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:29.986 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.984+0000 7f75d17fa640 10 monclient: dump: 2024-03-25T18:16:29.987 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-03-25T18:16:29.987 INFO:teuthology.orchestra.run.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:29.987 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:29.987 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:29.987 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:29.987 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:29.987 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:29.987 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:29.987 INFO:teuthology.orchestra.run.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:29.987 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-25T18:16:29.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.984+0000 7f75d17fa640 1 -- 172.21.15.122:0/2973724630 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f75cc0052e0 con 0x7f75b800c970 2024-03-25T18:16:29.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.984+0000 7f75d2ffd640 4 set_mon_vals no callback set 2024-03-25T18:16:29.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.984+0000 7f75d17fa640 1 -- 172.21.15.122:0/2973724630 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1098393589 0 0) 0x7f75cc0067b0 con 0x7f75b800c970 2024-03-25T18:16:29.987 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.984+0000 7f75d17fa640 10 client.4113.objecter ms_dispatch 0x7f75d4000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:16:29.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.984+0000 7f75d17fa640 3 client.4113.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:16:29.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.984+0000 7f75d17fa640 3 client.4113.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:16:29.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.984+0000 7f75d17fa640 20 client.4113.objecter dump_active .. 0 homeless 2024-03-25T18:16:29.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.984+0000 7f75d17fa640 1 -- 172.21.15.122:0/2973724630 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f75cc006ba0 con 0x7f75b800c970 2024-03-25T18:16:29.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.984+0000 7f75d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:29.988 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:29.984+0000 7f75d17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:30.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.059+0000 7f75daff4640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:16:30.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.059+0000 7f75daff4640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:16:30.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.059+0000 7f75daff4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:30.060 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.059+0000 7f75daff4640 1 -- 172.21.15.122:0/2973724630 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f75d41c9850 con 0x7f75b800c970 2024-03-25T18:16:30.171 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:30.170+0000 7fea77013040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:16:30.227 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:30.226+0000 7fea77013040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:16:30.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.237+0000 7f75d17fa640 1 -- 172.21.15.122:0/2973724630 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f75cc005670 con 0x7f75b800c970 2024-03-25T18:16:30.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.237+0000 7f75d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:16:30.238 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.237+0000 7f75d17fa640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:16:30.238 INFO:teuthology.orchestra.run.smithi122.stderr:adjusted tunables profile to default 2024-03-25T18:16:30.239 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.238+0000 7f75daff4640 10 librados: watch_flush enter 2024-03-25T18:16:30.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.238+0000 7f75daff4640 10 librados: watch_flush exit 2024-03-25T18:16:30.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.238+0000 7f75daff4640 20 client.4113.objecter shutdown clearing up homeless session... 2024-03-25T18:16:30.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.238+0000 7f75daff4640 10 client.4113.objecter successfully canceled tick 2024-03-25T18:16:30.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.238+0000 7f75daff4640 10 monclient: shutdown 2024-03-25T18:16:30.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.238+0000 7f75daff4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:30.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.238+0000 7f75daff4640 1 -- 172.21.15.122:0/2973724630 >> v1:172.21.15.122:6789/0 conn(0x7f75b800c970 legacy=0x7f75d41c9140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:30.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.238+0000 7f75daff4640 1 -- 172.21.15.122:0/2973724630 shutdown_connections 2024-03-25T18:16:30.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.238+0000 7f75daff4640 1 -- 172.21.15.122:0/2973724630 >> 172.21.15.122:0/2973724630 conn(0x7f75d40668e0 msgr2=0x7f75d40b6f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:30.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.238+0000 7f75daff4640 1 -- 172.21.15.122:0/2973724630 shutdown_connections 2024-03-25T18:16:30.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.238+0000 7f75daff4640 1 -- 172.21.15.122:0/2973724630 wait complete. 2024-03-25T18:16:30.240 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:30.238+0000 7f75daff4640 1 librados: shutdown 2024-03-25T18:16:30.246 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:30.246+0000 7f4b797b8040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:16:30.251 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:16:30.251 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:16:30.251 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:16:30.251 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:16:30.280 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:16:30.280 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:16:30.302 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:30.302+0000 7f4b797b8040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:16:30.349 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:16:30.349 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:16:30.416 DEBUG:teuthology.orchestra.run.smithi122:> set -ex 2024-03-25T18:16:30.416 DEBUG:teuthology.orchestra.run.smithi122:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:16:30.482 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T18:16:30.482 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:16:30.511 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T18:16:30.511 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:16:30.578 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T18:16:30.578 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:16:30.647 DEBUG:teuthology.orchestra.run.smithi204:> set -ex 2024-03-25T18:16:30.647 DEBUG:teuthology.orchestra.run.smithi204:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:16:30.714 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 7303eaf0-b038-4c4f-94f6-0c358667adc1 0 2024-03-25T18:16:30.790 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:30.789+0000 7fea77013040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:16:30.842 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:30.841+0000 7fea77013040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:16:30.854 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.853+0000 7fdd253c8640 10 monclient: get_monmap_and_config 2024-03-25T18:16:30.854 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.853+0000 7fdd253c8640 10 monclient: build_initial_monmap 2024-03-25T18:16:30.854 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.853+0000 7fdd253c8640 10 monclient: monmap: 2024-03-25T18:16:30.854 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-25T18:16:30.854 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:30.854 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:16:30.854702+0000 2024-03-25T18:16:30.854 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:16:30.854702+0000 2024-03-25T18:16:30.854 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:30.854 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:30.854 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:30.855 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:16:30.855 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:16:30.855 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:30.856 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.854+0000 7fdd253c8640 1 Processor -- start 2024-03-25T18:16:30.856 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.854+0000 7fdd253c8640 1 -- start start 2024-03-25T18:16:30.856 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.854+0000 7fdd253c8640 10 monclient: init 2024-03-25T18:16:30.856 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.855+0000 7fdd253c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:30.856 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.855+0000 7fdd253c8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:30.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.855+0000 7fdd253c8640 10 monclient(hunting): picked mon.noname-c con 0x7fdd2006caa0 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:30.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.855+0000 7fdd253c8640 10 monclient(hunting): picked mon.noname-b con 0x7fdd200ae1f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:30.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.855+0000 7fdd253c8640 10 monclient(hunting): picked mon.noname-a con 0x7fdd2013fc00 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:30.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.855+0000 7fdd253c8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd2006ee90 con 0x7fdd2013fc00 2024-03-25T18:16:30.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.855+0000 7fdd253c8640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd200adda0 con 0x7fdd2006caa0 2024-03-25T18:16:30.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.855+0000 7fdd253c8640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd200729f0 con 0x7fdd200ae1f0 2024-03-25T18:16:30.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.855+0000 7fdd253c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:30.857 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.855+0000 7fdd253c8640 10 monclient(hunting): authenticate will time out at 7275.004800s 2024-03-25T18:16:30.857 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:30.855+0000 7f4b797b8040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:16:30.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.855+0000 7fdd1effd640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fdd2006caa0 0x7fdd2013f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:33696/0 (socket says 172.21.15.204:33696) 2024-03-25T18:16:30.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.855+0000 7fdd1effd640 1 -- 172.21.15.204:0/1199471309 learned_addr learned my addr 172.21.15.204:0/1199471309 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-25T18:16:30.858 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 1 -- 172.21.15.204:0/1199471309 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735464191 0 0) 0x7fdd200729f0 con 0x7fdd200ae1f0 2024-03-25T18:16:30.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:30.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:30.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:30.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 10 monclient(hunting): my global_id is 4115 2024-03-25T18:16:30.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 1 -- 172.21.15.204:0/1199471309 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd08003520 con 0x7fdd200ae1f0 2024-03-25T18:16:30.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 1 -- 172.21.15.204:0/1199471309 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 895075285 0 0) 0x7fdd200adda0 con 0x7fdd2006caa0 2024-03-25T18:16:30.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:30.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:30.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:30.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:16:30.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 1 -- 172.21.15.204:0/1199471309 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd200729f0 con 0x7fdd2006caa0 2024-03-25T18:16:30.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 1 -- 172.21.15.204:0/1199471309 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3561454460 0 0) 0x7fdd2006ee90 con 0x7fdd2013fc00 2024-03-25T18:16:30.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:30.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:30.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:30.859 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:16:30.860 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.858+0000 7fdd1dffb640 1 -- 172.21.15.204:0/1199471309 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd200adda0 con 0x7fdd2013fc00 2024-03-25T18:16:30.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.861+0000 7fdd1dffb640 1 -- 172.21.15.204:0/1199471309 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1580309966 0 0) 0x7fdd200adda0 con 0x7fdd2013fc00 2024-03-25T18:16:30.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.861+0000 7fdd1dffb640 1 -- 172.21.15.204:0/1199471309 >> v1:172.21.15.122:6790/0 conn(0x7fdd200ae1f0 legacy=0x7fdd20145930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:30.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.861+0000 7fdd1dffb640 1 -- 172.21.15.204:0/1199471309 >> v1:172.21.15.204:6789/0 conn(0x7fdd2006caa0 legacy=0x7fdd2013f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:30.862 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.861+0000 7fdd1dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:30.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.861+0000 7fdd1dffb640 1 monclient: found mon.noname-a 2024-03-25T18:16:30.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.861+0000 7fdd1dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:30.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.861+0000 7fdd1dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:16:30.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.861+0000 7fdd1dffb640 1 -- 172.21.15.204:0/1199471309 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd20072b80 con 0x7fdd2013fc00 2024-03-25T18:16:30.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.861+0000 7fdd1dffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:30.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.861+0000 7fdd1dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:30.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.861+0000 7fdd1dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:30.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.861+0000 7fdd1dffb640 1 -- 172.21.15.204:0/1199471309 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fdd18001930 con 0x7fdd2013fc00 2024-03-25T18:16:30.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.861+0000 7fdd1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:30.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.861+0000 7fdd1dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:30.863 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.861+0000 7fdd1dffb640 10 monclient: dump: 2024-03-25T18:16:30.863 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:30.863 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:30.863 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:30.863 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:30.864 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:30.864 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:30.864 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:30.864 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:30.864 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:30.864 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:30.864 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.861+0000 7fdd253c8640 5 monclient: authenticate success, global_id 4116 2024-03-25T18:16:30.864 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.861+0000 7fdd253c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:30.866 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.863+0000 7fdd1dffb640 1 -- 172.21.15.204:0/1199471309 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd18001a90 con 0x7fdd2013fc00 2024-03-25T18:16:30.866 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.863+0000 7fdd1dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:30.866 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.863+0000 7fdd1dffb640 1 -- 172.21.15.204:0/1199471309 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fdd18001d40 con 0x7fdd2013fc00 2024-03-25T18:16:30.866 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.863+0000 7fdd1dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:30.866 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.863+0000 7fdd1dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:30.866 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.863+0000 7fdd1dffb640 10 monclient: dump: 2024-03-25T18:16:30.866 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:30.866 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:30.867 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:30.867 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:30.867 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:30.867 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:30.867 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:30.867 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:30.867 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:30.867 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:30.867 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.863+0000 7fdd253c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:30.867 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.863+0000 7fdd253c8640 4 set_mon_vals no callback set 2024-03-25T18:16:30.867 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.864+0000 7fdd253c8640 10 monclient: shutdown 2024-03-25T18:16:30.867 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.864+0000 7fdd253c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:30.867 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.864+0000 7fdd253c8640 1 -- 172.21.15.204:0/1199471309 >> v1:172.21.15.122:6789/0 conn(0x7fdd2013fc00 legacy=0x7fdd20146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:30.867 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.864+0000 7fdd253c8640 1 -- 172.21.15.204:0/1199471309 shutdown_connections 2024-03-25T18:16:30.867 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.864+0000 7fdd253c8640 1 -- 172.21.15.204:0/1199471309 >> 172.21.15.204:0/1199471309 conn(0x7fdd2006bcb0 msgr2=0x7fdd2006c0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:30.867 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.864+0000 7fdd253c8640 1 -- 172.21.15.204:0/1199471309 shutdown_connections 2024-03-25T18:16:30.868 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.864+0000 7fdd253c8640 1 -- 172.21.15.204:0/1199471309 wait complete. 2024-03-25T18:16:30.868 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.865+0000 7fdd253c8640 10 monclient: build_initial_monmap 2024-03-25T18:16:30.868 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.865+0000 7fdd253c8640 10 monclient: monmap: 2024-03-25T18:16:30.868 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-25T18:16:30.868 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:30.868 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-03-25T18:16:30.868 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-03-25T18:16:30.868 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:30.868 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-25T18:16:30.868 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:30.868 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:16:30.868 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:16:30.868 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:30.868 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.865+0000 7fdd253c8640 1 librados: starting msgr at 2024-03-25T18:16:30.868 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.865+0000 7fdd253c8640 1 librados: starting objecter 2024-03-25T18:16:30.868 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.865+0000 7fdd253c8640 1 Processor -- start 2024-03-25T18:16:30.869 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.865+0000 7fdd253c8640 1 -- start start 2024-03-25T18:16:30.869 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.865+0000 7fdd253c8640 1 librados: setting wanted keys 2024-03-25T18:16:30.869 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.865+0000 7fdd253c8640 1 librados: calling monclient init 2024-03-25T18:16:30.869 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.865+0000 7fdd253c8640 10 monclient: init 2024-03-25T18:16:30.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.866+0000 7fdd253c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:30.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.866+0000 7fdd253c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:30.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.866+0000 7fdd253c8640 10 monclient(hunting): picked mon.noname-c con 0x7fdd2006caa0 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:30.870 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.866+0000 7fdd253c8640 10 monclient(hunting): picked mon.noname-b con 0x7fdd200ae1f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:30.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.866+0000 7fdd253c8640 10 monclient(hunting): picked mon.noname-a con 0x7fdd2013fc00 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:30.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.866+0000 7fdd253c8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd201d7510 con 0x7fdd2013fc00 2024-03-25T18:16:30.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.866+0000 7fdd253c8640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd20137000 con 0x7fdd200ae1f0 2024-03-25T18:16:30.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.866+0000 7fdd253c8640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd20135980 con 0x7fdd2006caa0 2024-03-25T18:16:30.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.866+0000 7fdd253c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:30.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.866+0000 7fdd253c8640 10 monclient(hunting): authenticate will time out at 7275.016234s 2024-03-25T18:16:30.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.866+0000 7fdd1f7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fdd2013fc00 0x7fdd201d47c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.204:37564/0 (socket says 172.21.15.204:37564) 2024-03-25T18:16:30.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.866+0000 7fdd1f7fe640 1 -- 172.21.15.204:0/740001470 learned_addr learned my addr 172.21.15.204:0/740001470 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-25T18:16:30.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.867+0000 7fdd077fe640 10 client.?.objecter ms_handle_connect 0x7fdd2013fc00 2024-03-25T18:16:30.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.867+0000 7fdd077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:30.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.869+0000 7fdd077fe640 1 -- 172.21.15.204:0/740001470 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314006188 0 0) 0x7fdd201d7510 con 0x7fdd2013fc00 2024-03-25T18:16:30.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.869+0000 7fdd077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:30.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.869+0000 7fdd077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:30.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.869+0000 7fdd077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:30.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.869+0000 7fdd077fe640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:16:30.871 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.869+0000 7fdd077fe640 1 -- 172.21.15.204:0/740001470 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd0c004eb0 con 0x7fdd2013fc00 2024-03-25T18:16:30.875 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.874+0000 7fdd077fe640 1 -- 172.21.15.204:0/740001470 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1801699705 0 0) 0x7fdd0c004eb0 con 0x7fdd2013fc00 2024-03-25T18:16:30.875 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.874+0000 7fdd077fe640 1 -- 172.21.15.204:0/740001470 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd201d7510 con 0x7fdd2013fc00 2024-03-25T18:16:30.875 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.874+0000 7fdd077fe640 1 -- 172.21.15.204:0/740001470 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fdd18001ea0 con 0x7fdd2013fc00 2024-03-25T18:16:30.875 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.874+0000 7fdd077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:30.875 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.874+0000 7fdd077fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:30.875 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.874+0000 7fdd077fe640 10 monclient(hunting): dump: 2024-03-25T18:16:30.875 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:30.875 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:30.875 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:30.875 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:30.876 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:30.876 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:30.876 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:30.876 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:30.876 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:30.876 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:30.876 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.874+0000 7fdd077fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:30.878 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd077fe640 1 -- 172.21.15.204:0/740001470 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1506054653 0 0) 0x7fdd201d7510 con 0x7fdd2013fc00 2024-03-25T18:16:30.878 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd077fe640 1 -- 172.21.15.204:0/740001470 >> v1:172.21.15.122:6790/0 conn(0x7fdd2006caa0 legacy=0x7fdd20135270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:30.878 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd077fe640 1 -- 172.21.15.204:0/740001470 >> v1:172.21.15.204:6789/0 conn(0x7fdd200ae1f0 legacy=0x7fdd201d7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:30.878 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd077fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:30.878 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd077fe640 1 monclient: found mon.a 2024-03-25T18:16:30.878 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:30.878 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd077fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:30.878 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd077fe640 1 -- 172.21.15.204:0/740001470 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd201d7740 con 0x7fdd2013fc00 2024-03-25T18:16:30.878 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd077fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:30.878 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd077fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:30.879 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:30.879 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd253c8640 5 monclient: authenticate success, global_id 4119 2024-03-25T18:16:30.879 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd253c8640 10 monclient: _renew_subs 2024-03-25T18:16:30.879 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd253c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:30.879 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd253c8640 1 -- 172.21.15.204:0/740001470 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd201d8370 con 0x7fdd2013fc00 2024-03-25T18:16:30.879 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd253c8640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:30.879 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd253c8640 10 monclient: _renew_subs 2024-03-25T18:16:30.879 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd253c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:30.879 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd253c8640 1 -- 172.21.15.204:0/740001470 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd201d50b0 con 0x7fdd2013fc00 2024-03-25T18:16:30.879 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.877+0000 7fdd253c8640 1 librados: init done 2024-03-25T18:16:30.880 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.878+0000 7fdd253c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:30.880 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.878+0000 7fdd253c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:30.880 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.878+0000 7fdd253c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:30.880 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.878+0000 7fdd253c8640 1 -- 172.21.15.204:0/740001470 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd200682a0 con 0x7fdd2013fc00 2024-03-25T18:16:30.888 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:30.888+0000 7fea77013040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:16:30.914 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:30.913+0000 7f4b797b8040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:16:30.943 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:30.942+0000 7fea77013040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:16:30.959 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:30.959+0000 7f4b797b8040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:16:30.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.977+0000 7fdd077fe640 1 -- 172.21.15.204:0/740001470 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd18002000 con 0x7fdd2013fc00 2024-03-25T18:16:30.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.977+0000 7fdd077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:30.978 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.977+0000 7fdd077fe640 1 -- 172.21.15.204:0/740001470 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fdd18003dc0 con 0x7fdd2013fc00 2024-03-25T18:16:30.978 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.977+0000 7fdd077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:30.978 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.977+0000 7fdd077fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:30.978 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.977+0000 7fdd077fe640 10 monclient: dump: 2024-03-25T18:16:30.978 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:30.978 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:30.978 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:30.978 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:30.978 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:30.978 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:30.978 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:30.978 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:30.978 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:30.979 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:30.979 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.977+0000 7fdd077fe640 1 -- 172.21.15.204:0/740001470 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fdd18004370 con 0x7fdd2013fc00 2024-03-25T18:16:30.979 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.977+0000 7fdd1d7fa640 4 set_mon_vals no callback set 2024-03-25T18:16:30.979 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.978+0000 7fdd077fe640 1 -- 172.21.15.204:0/740001470 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 551313507 0 0) 0x7fdd18006990 con 0x7fdd2013fc00 2024-03-25T18:16:30.979 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.978+0000 7fdd077fe640 10 client.4119.objecter ms_dispatch 0x7fdd20000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:16:30.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.978+0000 7fdd077fe640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:16:30.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.978+0000 7fdd077fe640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:16:30.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.978+0000 7fdd077fe640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T18:16:30.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.978+0000 7fdd077fe640 1 -- 172.21.15.204:0/740001470 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fdd18005b00 con 0x7fdd2013fc00 2024-03-25T18:16:30.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.978+0000 7fdd077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:30.980 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:30.978+0000 7fdd077fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:30.986 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:30.986+0000 7fea77013040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:16:31.016 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:31.015+0000 7f4b797b8040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:16:31.050 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.050+0000 7fdd253c8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7303eaf0-b038-4c4f-94f6-0c358667adc1", "id": 0}] 2024-03-25T18:16:31.051 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.050+0000 7fdd253c8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7303eaf0-b038-4c4f-94f6-0c358667adc1", "id": 0}] 2024-03-25T18:16:31.051 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.050+0000 7fdd253c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:31.051 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.050+0000 7fdd253c8640 1 -- 172.21.15.204:0/740001470 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7303eaf0-b038-4c4f-94f6-0c358667adc1", "id": 0} v 0) v1 -- 0x7fdd2004f730 con 0x7fdd2013fc00 2024-03-25T18:16:31.060 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:31.059+0000 7f4b797b8040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:16:31.103 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:31.103+0000 7fea77013040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:16:31.132 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.131+0000 7fdd077fe640 1 -- 172.21.15.204:0/740001470 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7303eaf0-b038-4c4f-94f6-0c358667adc1", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3458819143 0 732707515) 0x7fdd18005d40 con 0x7fdd2013fc00 2024-03-25T18:16:31.132 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.131+0000 7fdd077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7303eaf0-b038-4c4f-94f6-0c358667adc1", "id": 0}] 2024-03-25T18:16:31.132 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.131+0000 7fdd077fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:31.132 INFO:teuthology.orchestra.run.smithi204.stdout:0 2024-03-25T18:16:31.133 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.132+0000 7fdd253c8640 10 librados: watch_flush enter 2024-03-25T18:16:31.133 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.132+0000 7fdd253c8640 10 librados: watch_flush exit 2024-03-25T18:16:31.134 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.132+0000 7fdd253c8640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T18:16:31.134 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.132+0000 7fdd253c8640 10 client.4119.objecter successfully canceled tick 2024-03-25T18:16:31.134 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.132+0000 7fdd253c8640 10 monclient: shutdown 2024-03-25T18:16:31.134 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.132+0000 7fdd253c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:31.134 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.132+0000 7fdd253c8640 1 -- 172.21.15.204:0/740001470 >> v1:172.21.15.122:6789/0 conn(0x7fdd2013fc00 legacy=0x7fdd201d47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:31.134 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.132+0000 7fdd253c8640 1 -- 172.21.15.204:0/740001470 shutdown_connections 2024-03-25T18:16:31.134 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.132+0000 7fdd253c8640 1 -- 172.21.15.204:0/740001470 >> 172.21.15.204:0/740001470 conn(0x7fdd2006bcb0 msgr2=0x7fdd200b06e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:31.134 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.132+0000 7fdd253c8640 1 -- 172.21.15.204:0/740001470 shutdown_connections 2024-03-25T18:16:31.134 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.132+0000 7fdd253c8640 1 -- 172.21.15.204:0/740001470 wait complete. 2024-03-25T18:16:31.134 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.132+0000 7fdd253c8640 1 librados: shutdown 2024-03-25T18:16:31.143 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 97ebd37c-5267-47ee-a84d-80a6e7444cde 1 2024-03-25T18:16:31.179 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:31.178+0000 7fea77013040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:16:31.183 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:31.182+0000 7f4b797b8040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:16:31.224 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:31.223+0000 7fea77013040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:16:31.243 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.242+0000 7f684c39d640 10 monclient: get_monmap_and_config 2024-03-25T18:16:31.243 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.242+0000 7f684c39d640 10 monclient: build_initial_monmap 2024-03-25T18:16:31.243 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.242+0000 7f684c39d640 10 monclient: monmap: 2024-03-25T18:16:31.243 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-25T18:16:31.243 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:31.243 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:16:31.243538+0000 2024-03-25T18:16:31.243 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:16:31.243538+0000 2024-03-25T18:16:31.243 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:31.243 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:31.243 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:31.243 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:16:31.244 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:16:31.244 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:31.245 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.243+0000 7f684c39d640 1 Processor -- start 2024-03-25T18:16:31.245 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.243+0000 7f684c39d640 1 -- start start 2024-03-25T18:16:31.245 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.243+0000 7f684c39d640 10 monclient: init 2024-03-25T18:16:31.245 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.243+0000 7f684c39d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:31.245 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.243+0000 7f684c39d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:31.245 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.243+0000 7f684c39d640 10 monclient(hunting): picked mon.noname-a con 0x7f68440b4700 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:31.245 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.243+0000 7f684c39d640 10 monclient(hunting): picked mon.noname-b con 0x7f6844142a20 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:31.245 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.243+0000 7f684c39d640 10 monclient(hunting): picked mon.noname-c con 0x7f68440a96b0 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:31.245 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.244+0000 7f684c39d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68440a9d60 con 0x7f68440b4700 2024-03-25T18:16:31.246 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.244+0000 7f684c39d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f684413ff10 con 0x7f68440a96b0 2024-03-25T18:16:31.246 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.244+0000 7f684c39d640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68440aa1f0 con 0x7f6844142a20 2024-03-25T18:16:31.246 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.244+0000 7f684c39d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:31.246 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.244+0000 7f684c39d640 10 monclient(hunting): authenticate will time out at 7275.393727s 2024-03-25T18:16:31.246 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.244+0000 7f684a139640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f68440b4700 0x7f6844142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.204:37568/0 (socket says 172.21.15.204:37568) 2024-03-25T18:16:31.246 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.244+0000 7f684a139640 1 -- 172.21.15.204:0/1579498131 learned_addr learned my addr 172.21.15.204:0/1579498131 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-25T18:16:31.246 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.246+0000 7f6849137640 1 -- 172.21.15.204:0/1579498131 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893580358 0 0) 0x7f684413ff10 con 0x7f68440a96b0 2024-03-25T18:16:31.247 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.246+0000 7f6849137640 10 monclient(hunting): _negotiate 2024-03-25T18:16:31.247 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.246+0000 7f6849137640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:31.247 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.246+0000 7f6849137640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:31.247 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.246+0000 7f6849137640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:16:31.247 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.246+0000 7f6849137640 1 -- 172.21.15.204:0/1579498131 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6830003520 con 0x7f68440a96b0 2024-03-25T18:16:31.248 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.247+0000 7f6849137640 1 -- 172.21.15.204:0/1579498131 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101758906 0 0) 0x7f68440a9d60 con 0x7f68440b4700 2024-03-25T18:16:31.248 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.247+0000 7f6849137640 10 monclient(hunting): _negotiate 2024-03-25T18:16:31.248 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.247+0000 7f6849137640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:31.248 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.247+0000 7f6849137640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:31.248 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.247+0000 7f6849137640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:16:31.248 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.247+0000 7f6849137640 1 -- 172.21.15.204:0/1579498131 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f684413ff10 con 0x7f68440b4700 2024-03-25T18:16:31.248 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.247+0000 7f6849137640 1 -- 172.21.15.204:0/1579498131 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3339710255 0 0) 0x7f68440aa1f0 con 0x7f6844142a20 2024-03-25T18:16:31.248 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.247+0000 7f6849137640 10 monclient(hunting): _negotiate 2024-03-25T18:16:31.248 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.247+0000 7f6849137640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:31.248 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.247+0000 7f6849137640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:31.248 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.247+0000 7f6849137640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:16:31.248 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.247+0000 7f6849137640 1 -- 172.21.15.204:0/1579498131 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68440a9d60 con 0x7f6844142a20 2024-03-25T18:16:31.251 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f6849137640 1 -- 172.21.15.204:0/1579498131 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1481742128 0 0) 0x7f6830003520 con 0x7f68440a96b0 2024-03-25T18:16:31.251 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f6849137640 1 -- 172.21.15.204:0/1579498131 >> v1:172.21.15.122:6790/0 conn(0x7f6844142a20 legacy=0x7f684413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:31.252 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f6849137640 1 -- 172.21.15.204:0/1579498131 >> v1:172.21.15.122:6789/0 conn(0x7f68440b4700 legacy=0x7f6844142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:31.252 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f6849137640 10 monclient: _finish_hunting 0 2024-03-25T18:16:31.252 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f6849137640 1 monclient: found mon.noname-c 2024-03-25T18:16:31.252 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f6849137640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:31.252 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f6849137640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:16:31.252 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f6849137640 1 -- 172.21.15.204:0/1579498131 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68440b4ae0 con 0x7f68440a96b0 2024-03-25T18:16:31.252 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f6849137640 10 monclient: _finish_auth 0 2024-03-25T18:16:31.252 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f6849137640 10 monclient: _check_auth_tickets 2024-03-25T18:16:31.252 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f6849137640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:31.252 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f6849137640 1 -- 172.21.15.204:0/1579498131 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f683c001a10 con 0x7f68440a96b0 2024-03-25T18:16:31.252 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f684c39d640 5 monclient: authenticate success, global_id 4120 2024-03-25T18:16:31.252 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f684c39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:31.252 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f6849137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:31.252 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f6849137640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:31.253 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f6849137640 10 monclient: dump: 2024-03-25T18:16:31.253 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:31.253 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:31.253 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.253 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.253 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:31.253 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:31.253 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:31.253 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:31.253 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:31.253 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:31.253 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.250+0000 7f684c39d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:31.257 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:31.257+0000 7f4b797b8040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:16:31.277 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:31.276+0000 7fea77013040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:16:31.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.276+0000 7f6849137640 1 -- 172.21.15.204:0/1579498131 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f683c001b70 con 0x7f68440a96b0 2024-03-25T18:16:31.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.276+0000 7f6849137640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:31.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.276+0000 7f6849137640 1 -- 172.21.15.204:0/1579498131 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f683c001f80 con 0x7f68440a96b0 2024-03-25T18:16:31.277 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.276+0000 7f6849137640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:31.278 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.276+0000 7f6849137640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:31.278 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.276+0000 7f6849137640 10 monclient: dump: 2024-03-25T18:16:31.278 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:31.278 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:31.278 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.278 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.278 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:31.278 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:31.278 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:31.278 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:31.278 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:31.278 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:31.278 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.276+0000 7f684c39d640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:31.279 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.276+0000 7f684c39d640 4 set_mon_vals no callback set 2024-03-25T18:16:31.281 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.277+0000 7f684c39d640 10 monclient: shutdown 2024-03-25T18:16:31.281 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.277+0000 7f684c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:31.281 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.277+0000 7f684c39d640 1 -- 172.21.15.204:0/1579498131 >> v1:172.21.15.204:6789/0 conn(0x7f68440a96b0 legacy=0x7f684413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:31.281 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.277+0000 7f684c39d640 1 -- 172.21.15.204:0/1579498131 shutdown_connections 2024-03-25T18:16:31.281 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.277+0000 7f684c39d640 1 -- 172.21.15.204:0/1579498131 >> 172.21.15.204:0/1579498131 conn(0x7f68440668e0 msgr2=0x7f6844066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:31.281 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.277+0000 7f684c39d640 1 -- 172.21.15.204:0/1579498131 shutdown_connections 2024-03-25T18:16:31.281 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.277+0000 7f684c39d640 1 -- 172.21.15.204:0/1579498131 wait complete. 2024-03-25T18:16:31.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.278+0000 7f684c39d640 10 monclient: build_initial_monmap 2024-03-25T18:16:31.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.278+0000 7f684c39d640 10 monclient: monmap: 2024-03-25T18:16:31.282 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-25T18:16:31.282 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:31.282 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-03-25T18:16:31.282 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-03-25T18:16:31.282 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:31.282 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-25T18:16:31.282 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:31.282 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:16:31.282 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:16:31.282 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:31.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.278+0000 7f684c39d640 1 librados: starting msgr at 2024-03-25T18:16:31.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.278+0000 7f684c39d640 1 librados: starting objecter 2024-03-25T18:16:31.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.278+0000 7f684c39d640 1 Processor -- start 2024-03-25T18:16:31.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.278+0000 7f684c39d640 1 -- start start 2024-03-25T18:16:31.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.278+0000 7f684c39d640 1 librados: setting wanted keys 2024-03-25T18:16:31.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.278+0000 7f684c39d640 1 librados: calling monclient init 2024-03-25T18:16:31.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.278+0000 7f684c39d640 10 monclient: init 2024-03-25T18:16:31.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.279+0000 7f684c39d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:31.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.279+0000 7f684c39d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:31.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.279+0000 7f684c39d640 10 monclient(hunting): picked mon.noname-b con 0x7f68440a96b0 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:31.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.279+0000 7f684c39d640 10 monclient(hunting): picked mon.noname-a con 0x7f68440b4700 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:31.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.279+0000 7f684c39d640 10 monclient(hunting): picked mon.noname-c con 0x7f6844142a20 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:31.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.279+0000 7f684c39d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68440abad0 con 0x7f68440b4700 2024-03-25T18:16:31.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.279+0000 7f684c39d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68440abdb0 con 0x7f68440a96b0 2024-03-25T18:16:31.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.279+0000 7f684c39d640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68440ac090 con 0x7f6844142a20 2024-03-25T18:16:31.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.279+0000 7f684c39d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:31.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.279+0000 7f684c39d640 10 monclient(hunting): authenticate will time out at 7275.429184s 2024-03-25T18:16:31.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.280+0000 7f6849938640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f68440b4700 0x7f68441396b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.204:37582/0 (socket says 172.21.15.204:37582) 2024-03-25T18:16:31.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.280+0000 7f6849938640 1 -- 172.21.15.204:0/1962567769 learned_addr learned my addr 172.21.15.204:0/1962567769 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-25T18:16:31.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.280+0000 7f683affd640 10 client.?.objecter ms_handle_connect 0x7f68440b4700 2024-03-25T18:16:31.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.280+0000 7f683affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:31.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.280+0000 7f683affd640 10 client.?.objecter ms_handle_connect 0x7f6844142a20 2024-03-25T18:16:31.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.280+0000 7f683affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:31.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.280+0000 7f683affd640 10 client.?.objecter ms_handle_connect 0x7f68440a96b0 2024-03-25T18:16:31.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.280+0000 7f683affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:31.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.282+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3434135746 0 0) 0x7f68440abad0 con 0x7f68440b4700 2024-03-25T18:16:31.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.282+0000 7f683affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:31.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.282+0000 7f683affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:31.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.282+0000 7f683affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:31.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.282+0000 7f683affd640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:16:31.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.282+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6828004eb0 con 0x7f68440b4700 2024-03-25T18:16:31.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.282+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968158470 0 0) 0x7f68440ac090 con 0x7f6844142a20 2024-03-25T18:16:31.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.282+0000 7f683affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:31.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.282+0000 7f683affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:31.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.282+0000 7f683affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:31.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.282+0000 7f683affd640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:16:31.286 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.282+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68440abad0 con 0x7f6844142a20 2024-03-25T18:16:31.286 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.283+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500135031 0 0) 0x7f68440abdb0 con 0x7f68440a96b0 2024-03-25T18:16:31.286 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.283+0000 7f683affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:31.286 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.283+0000 7f683affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:31.286 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.283+0000 7f683affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:31.286 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.283+0000 7f683affd640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:16:31.286 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.283+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68440ac090 con 0x7f68440a96b0 2024-03-25T18:16:31.290 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.289+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 142363027 0 0) 0x7f6828004eb0 con 0x7f68440b4700 2024-03-25T18:16:31.290 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.289+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68440abdb0 con 0x7f68440b4700 2024-03-25T18:16:31.290 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.289+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f6834001a10 con 0x7f68440b4700 2024-03-25T18:16:31.290 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.289+0000 7f683affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:31.290 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.289+0000 7f683affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:31.290 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.289+0000 7f683affd640 10 monclient(hunting): dump: 2024-03-25T18:16:31.290 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:31.290 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:31.290 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.290 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.290 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:31.290 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:31.290 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:31.290 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:31.291 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:31.291 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:31.291 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.289+0000 7f683affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:31.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2358367146 0 0) 0x7f68440abdb0 con 0x7f68440b4700 2024-03-25T18:16:31.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 >> v1:172.21.15.122:6790/0 conn(0x7f6844142a20 legacy=0x7f68441e2080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:31.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 >> v1:172.21.15.204:6789/0 conn(0x7f68440a96b0 legacy=0x7f68440ac4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:31.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f683affd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:31.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f683affd640 1 monclient: found mon.a 2024-03-25T18:16:31.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f683affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:31.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f683affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:31.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684413c220 con 0x7f68440b4700 2024-03-25T18:16:31.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f683affd640 10 monclient: _finish_auth 0 2024-03-25T18:16:31.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f683affd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:31.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f683affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:31.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f684c39d640 5 monclient: authenticate success, global_id 4125 2024-03-25T18:16:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f684c39d640 10 monclient: _renew_subs 2024-03-25T18:16:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f684c39d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f684c39d640 1 -- 172.21.15.204:0/1962567769 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f684413afa0 con 0x7f68440b4700 2024-03-25T18:16:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f684c39d640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f684c39d640 10 monclient: _renew_subs 2024-03-25T18:16:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f684c39d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f684c39d640 1 -- 172.21.15.204:0/1962567769 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f684413bd20 con 0x7f68440b4700 2024-03-25T18:16:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.293+0000 7f684c39d640 1 librados: init done 2024-03-25T18:16:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.294+0000 7f684c39d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.294+0000 7f684c39d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.294+0000 7f684c39d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:31.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.294+0000 7f684c39d640 1 -- 172.21.15.204:0/1962567769 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f684413b190 con 0x7f68440b4700 2024-03-25T18:16:31.308 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:31.308+0000 7f4b797b8040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:16:31.326 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:31.325+0000 7fea77013040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:16:31.361 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:31.361+0000 7f4b797b8040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:16:31.391 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.390+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6834001c70 con 0x7f68440b4700 2024-03-25T18:16:31.391 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.390+0000 7f683affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:31.391 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.390+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f6834003a60 con 0x7f68440b4700 2024-03-25T18:16:31.391 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.390+0000 7f683affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:31.391 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.390+0000 7f683affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:31.391 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.390+0000 7f683affd640 10 monclient: dump: 2024-03-25T18:16:31.391 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:31.391 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:31.391 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.391 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.391 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:31.391 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:31.391 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:31.392 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:31.392 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:31.392 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:31.392 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.390+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6834004010 con 0x7f68440b4700 2024-03-25T18:16:31.392 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.390+0000 7f6848936640 4 set_mon_vals no callback set 2024-03-25T18:16:31.392 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.391+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 3913089967 0 0) 0x7f6834006670 con 0x7f68440b4700 2024-03-25T18:16:31.392 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.391+0000 7f683affd640 10 client.4125.objecter ms_dispatch 0x7f6844000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:16:31.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.391+0000 7f683affd640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:16:31.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.391+0000 7f683affd640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:16:31.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.391+0000 7f683affd640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T18:16:31.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.391+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f68340069c0 con 0x7f68440b4700 2024-03-25T18:16:31.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.391+0000 7f683affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:31.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.391+0000 7f683affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:31.409 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:31.409+0000 7f4b797b8040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:16:31.464 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.463+0000 7f684c39d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "97ebd37c-5267-47ee-a84d-80a6e7444cde", "id": 1}] 2024-03-25T18:16:31.464 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.463+0000 7f684c39d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "97ebd37c-5267-47ee-a84d-80a6e7444cde", "id": 1}] 2024-03-25T18:16:31.464 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.463+0000 7f684c39d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:31.464 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.463+0000 7f684c39d640 1 -- 172.21.15.204:0/1962567769 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd new", "uuid": "97ebd37c-5267-47ee-a84d-80a6e7444cde", "id": 1} v 0) v1 -- 0x7f684404f730 con 0x7f68440b4700 2024-03-25T18:16:31.501 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.500+0000 7f683affd640 1 -- 172.21.15.204:0/1962567769 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "97ebd37c-5267-47ee-a84d-80a6e7444cde", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 1499586145 0 940483276) 0x7f68340057c0 con 0x7f68440b4700 2024-03-25T18:16:31.501 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.500+0000 7f683affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "97ebd37c-5267-47ee-a84d-80a6e7444cde", "id": 1}] 2024-03-25T18:16:31.501 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.500+0000 7f683affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:31.501 INFO:teuthology.orchestra.run.smithi204.stdout:1 2024-03-25T18:16:31.502 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.501+0000 7f684c39d640 10 librados: watch_flush enter 2024-03-25T18:16:31.502 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.501+0000 7f684c39d640 10 librados: watch_flush exit 2024-03-25T18:16:31.503 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.501+0000 7f684c39d640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T18:16:31.503 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.501+0000 7f684c39d640 10 client.4125.objecter successfully canceled tick 2024-03-25T18:16:31.503 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.501+0000 7f684c39d640 10 monclient: shutdown 2024-03-25T18:16:31.503 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.501+0000 7f684c39d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:31.503 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.501+0000 7f684c39d640 1 -- 172.21.15.204:0/1962567769 >> v1:172.21.15.122:6789/0 conn(0x7f68440b4700 legacy=0x7f68441396b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:31.503 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.501+0000 7f684c39d640 1 -- 172.21.15.204:0/1962567769 shutdown_connections 2024-03-25T18:16:31.503 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.501+0000 7f684c39d640 1 -- 172.21.15.204:0/1962567769 >> 172.21.15.204:0/1962567769 conn(0x7f68440668e0 msgr2=0x7f684413e490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:31.503 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.501+0000 7f684c39d640 1 -- 172.21.15.204:0/1962567769 shutdown_connections 2024-03-25T18:16:31.503 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.501+0000 7f684c39d640 1 -- 172.21.15.204:0/1962567769 wait complete. 2024-03-25T18:16:31.503 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.501+0000 7f684c39d640 1 librados: shutdown 2024-03-25T18:16:31.512 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new 188de57d-73fb-49be-84c4-92af3a291860 2 2024-03-25T18:16:31.652 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.651+0000 7f9b3100d640 10 monclient: get_monmap_and_config 2024-03-25T18:16:31.652 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.651+0000 7f9b3100d640 10 monclient: build_initial_monmap 2024-03-25T18:16:31.652 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.651+0000 7f9b3100d640 10 monclient: monmap: 2024-03-25T18:16:31.652 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-25T18:16:31.653 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:31.653 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:16:31.652873+0000 2024-03-25T18:16:31.653 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:16:31.652873+0000 2024-03-25T18:16:31.653 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:31.653 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:31.653 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:31.653 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:16:31.653 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:16:31.653 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:31.654 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.652+0000 7f9b3100d640 1 Processor -- start 2024-03-25T18:16:31.655 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.652+0000 7f9b3100d640 1 -- start start 2024-03-25T18:16:31.655 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.652+0000 7f9b3100d640 10 monclient: init 2024-03-25T18:16:31.655 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.653+0000 7f9b3100d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:31.655 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.653+0000 7f9b3100d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:31.655 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.653+0000 7f9b3100d640 10 monclient(hunting): picked mon.noname-c con 0x7f9b2c0b4700 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:31.655 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.653+0000 7f9b3100d640 10 monclient(hunting): picked mon.noname-b con 0x7f9b2c142a20 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:31.655 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.653+0000 7f9b3100d640 10 monclient(hunting): picked mon.noname-a con 0x7f9b2c0a96b0 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:31.655 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.653+0000 7f9b3100d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2c0a9d60 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.655 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.653+0000 7f9b3100d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2c13ff10 con 0x7f9b2c0b4700 2024-03-25T18:16:31.655 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.653+0000 7f9b3100d640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2c0aa1f0 con 0x7f9b2c142a20 2024-03-25T18:16:31.655 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.653+0000 7f9b3100d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:31.655 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.653+0000 7f9b3100d640 10 monclient(hunting): authenticate will time out at 7275.802920s 2024-03-25T18:16:31.655 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.654+0000 7f9b2ad9d640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f9b2c0b4700 0x7f9b2c142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:33724/0 (socket says 172.21.15.204:33724) 2024-03-25T18:16:31.655 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.654+0000 7f9b2ad9d640 1 -- 172.21.15.204:0/2007556782 learned_addr learned my addr 172.21.15.204:0/2007556782 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-25T18:16:31.669 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.668+0000 7f9b29d9b640 1 -- 172.21.15.204:0/2007556782 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3199573619 0 0) 0x7f9b2c0a9d60 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.669 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.668+0000 7f9b29d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:31.669 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.668+0000 7f9b29d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:31.669 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.668+0000 7f9b29d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:31.669 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.668+0000 7f9b29d9b640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:16:31.669 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.668+0000 7f9b29d9b640 1 -- 172.21.15.204:0/2007556782 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b14003520 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.672 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.671+0000 7f9b29d9b640 1 -- 172.21.15.204:0/2007556782 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3804578502 0 0) 0x7f9b14003520 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.672 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.671+0000 7f9b29d9b640 1 -- 172.21.15.204:0/2007556782 >> v1:172.21.15.122:6790/0 conn(0x7f9b2c142a20 legacy=0x7f9b2c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:31.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.671+0000 7f9b29d9b640 1 -- 172.21.15.204:0/2007556782 >> v1:172.21.15.204:6789/0 conn(0x7f9b2c0b4700 legacy=0x7f9b2c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:31.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.671+0000 7f9b29d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:16:31.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.671+0000 7f9b29d9b640 1 monclient: found mon.noname-a 2024-03-25T18:16:31.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.671+0000 7f9b29d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:31.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.671+0000 7f9b29d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:16:31.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.671+0000 7f9b29d9b640 1 -- 172.21.15.204:0/2007556782 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2c0b4ae0 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.671+0000 7f9b29d9b640 10 monclient: _finish_auth 0 2024-03-25T18:16:31.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.671+0000 7f9b29d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:16:31.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.671+0000 7f9b29d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:31.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.671+0000 7f9b3100d640 5 monclient: authenticate success, global_id 4128 2024-03-25T18:16:31.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.671+0000 7f9b3100d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:31.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.671+0000 7f9b29d9b640 1 -- 172.21.15.204:0/2007556782 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9b24001a10 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.671+0000 7f9b29d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:31.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.671+0000 7f9b29d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:31.673 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.671+0000 7f9b29d9b640 10 monclient: dump: 2024-03-25T18:16:31.674 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:31.674 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:31.674 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.674 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.674 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:31.674 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:31.674 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:31.674 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:31.674 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:31.674 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:31.674 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.672+0000 7f9b3100d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:31.676 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.675+0000 7f9b29d9b640 1 -- 172.21.15.204:0/2007556782 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b24001ca0 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.676 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.675+0000 7f9b29d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:31.676 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.675+0000 7f9b3100d640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:31.676 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.675+0000 7f9b3100d640 4 set_mon_vals no callback set 2024-03-25T18:16:31.677 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.676+0000 7f9b3100d640 10 monclient: shutdown 2024-03-25T18:16:31.677 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.676+0000 7f9b3100d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:31.677 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.676+0000 7f9b3100d640 1 -- 172.21.15.204:0/2007556782 >> v1:172.21.15.122:6789/0 conn(0x7f9b2c0a96b0 legacy=0x7f9b2c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:31.677 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.676+0000 7f9b3100d640 1 -- 172.21.15.204:0/2007556782 shutdown_connections 2024-03-25T18:16:31.677 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.676+0000 7f9b3100d640 1 -- 172.21.15.204:0/2007556782 >> 172.21.15.204:0/2007556782 conn(0x7f9b2c0668e0 msgr2=0x7f9b2c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:31.677 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.676+0000 7f9b3100d640 1 -- 172.21.15.204:0/2007556782 shutdown_connections 2024-03-25T18:16:31.677 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.676+0000 7f9b3100d640 1 -- 172.21.15.204:0/2007556782 wait complete. 2024-03-25T18:16:31.679 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.677+0000 7f9b3100d640 10 monclient: build_initial_monmap 2024-03-25T18:16:31.679 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.677+0000 7f9b3100d640 10 monclient: monmap: 2024-03-25T18:16:31.679 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-25T18:16:31.680 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:31.680 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-03-25T18:16:31.680 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-03-25T18:16:31.680 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:31.680 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-25T18:16:31.680 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:31.680 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:16:31.680 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:16:31.680 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:31.680 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.677+0000 7f9b3100d640 1 librados: starting msgr at 2024-03-25T18:16:31.680 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.677+0000 7f9b3100d640 1 librados: starting objecter 2024-03-25T18:16:31.680 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.677+0000 7f9b3100d640 1 Processor -- start 2024-03-25T18:16:31.680 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.678+0000 7f9b3100d640 1 -- start start 2024-03-25T18:16:31.680 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.678+0000 7f9b3100d640 1 librados: setting wanted keys 2024-03-25T18:16:31.680 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.678+0000 7f9b3100d640 1 librados: calling monclient init 2024-03-25T18:16:31.680 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.678+0000 7f9b3100d640 10 monclient: init 2024-03-25T18:16:31.681 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.678+0000 7f9b3100d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:31.681 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.678+0000 7f9b3100d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:31.681 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.678+0000 7f9b3100d640 10 monclient(hunting): picked mon.noname-a con 0x7f9b2c0a96b0 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:31.681 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.678+0000 7f9b3100d640 10 monclient(hunting): picked mon.noname-b con 0x7f9b2c0b4700 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:31.681 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.678+0000 7f9b3100d640 10 monclient(hunting): picked mon.noname-c con 0x7f9b2c142a20 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:31.681 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.678+0000 7f9b3100d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2c135790 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.681 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.678+0000 7f9b3100d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2c0ac9e0 con 0x7f9b2c0b4700 2024-03-25T18:16:31.681 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.678+0000 7f9b3100d640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b2c1dbe60 con 0x7f9b2c142a20 2024-03-25T18:16:31.681 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.678+0000 7f9b3100d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:31.681 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.678+0000 7f9b3100d640 10 monclient(hunting): authenticate will time out at 7275.828271s 2024-03-25T18:16:31.681 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.678+0000 7f9b2ad9d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9b2c0a96b0 0x7f9b2c0ac2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.204:37592/0 (socket says 172.21.15.204:37592) 2024-03-25T18:16:31.681 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.678+0000 7f9b2ad9d640 1 -- 172.21.15.204:0/3766373321 learned_addr learned my addr 172.21.15.204:0/3766373321 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-25T18:16:31.681 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.679+0000 7f9b137fe640 10 client.?.objecter ms_handle_connect 0x7f9b2c0a96b0 2024-03-25T18:16:31.681 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.679+0000 7f9b137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:31.681 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.679+0000 7f9b137fe640 10 client.?.objecter ms_handle_connect 0x7f9b2c142a20 2024-03-25T18:16:31.682 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.679+0000 7f9b137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:31.682 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.679+0000 7f9b137fe640 10 client.?.objecter ms_handle_connect 0x7f9b2c0b4700 2024-03-25T18:16:31.682 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.679+0000 7f9b137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:31.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415064914 0 0) 0x7f9b2c1dbe60 con 0x7f9b2c142a20 2024-03-25T18:16:31.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:31.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:31.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:31.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:16:31.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b18004eb0 con 0x7f9b2c142a20 2024-03-25T18:16:31.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133177930 0 0) 0x7f9b2c135790 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:31.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:31.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:31.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:16:31.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b2c1dbe60 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 615722104 0 0) 0x7f9b2c0ac9e0 con 0x7f9b2c0b4700 2024-03-25T18:16:31.684 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:31.684 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:31.684 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:31.684 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:16:31.684 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.681+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b2c135790 con 0x7f9b2c0b4700 2024-03-25T18:16:31.686 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.685+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3617750361 0 0) 0x7f9b18004eb0 con 0x7f9b2c142a20 2024-03-25T18:16:31.686 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.685+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b2c0ac9e0 con 0x7f9b2c142a20 2024-03-25T18:16:31.686 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.686+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9b240019d0 con 0x7f9b2c142a20 2024-03-25T18:16:31.686 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.686+0000 7f9b137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:31.686 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.686+0000 7f9b137fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:31.686 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.686+0000 7f9b137fe640 10 monclient(hunting): dump: 2024-03-25T18:16:31.686 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:31.686 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:31.687 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.687 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.687 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:31.687 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:31.687 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:31.687 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:31.687 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:31.687 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:31.687 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.686+0000 7f9b137fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:31.688 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.687+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 408433352 0 0) 0x7f9b2c135790 con 0x7f9b2c0b4700 2024-03-25T18:16:31.688 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.687+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b18004eb0 con 0x7f9b2c0b4700 2024-03-25T18:16:31.688 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.687+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9b200017c0 con 0x7f9b2c0b4700 2024-03-25T18:16:31.688 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.687+0000 7f9b137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:31.688 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.687+0000 7f9b137fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:31.688 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.687+0000 7f9b137fe640 10 monclient(hunting): dump: 2024-03-25T18:16:31.688 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:31.688 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:31.688 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.688 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.688 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:31.689 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:31.689 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:31.689 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:31.689 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:31.689 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:31.689 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.688+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4033358337 0 0) 0x7f9b2c1dbe60 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.689 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.688+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b2c135790 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.690 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.688+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9b1c0018f0 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.690 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.688+0000 7f9b137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:31.690 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.688+0000 7f9b137fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:31.690 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.688+0000 7f9b137fe640 10 monclient(hunting): dump: 2024-03-25T18:16:31.690 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:31.690 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:31.690 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.690 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.690 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:31.690 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:31.690 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:31.690 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:31.690 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:31.690 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:31.693 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.692+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3262770476 0 0) 0x7f9b2c135790 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.693 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.692+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 >> v1:172.21.15.122:6790/0 conn(0x7f9b2c142a20 legacy=0x7f9b2c1db750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:31.693 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.692+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 >> v1:172.21.15.204:6789/0 conn(0x7f9b2c0b4700 legacy=0x7f9b2c1db040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:31.693 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.692+0000 7f9b137fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:31.693 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.692+0000 7f9b137fe640 1 monclient: found mon.a 2024-03-25T18:16:31.694 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.692+0000 7f9b137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:31.694 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.692+0000 7f9b137fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:31.694 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.692+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b2c0abe40 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.694 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.692+0000 7f9b137fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:31.694 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.692+0000 7f9b137fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:31.694 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.692+0000 7f9b137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:31.694 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.692+0000 7f9b3100d640 5 monclient: authenticate success, global_id 4131 2024-03-25T18:16:31.694 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.692+0000 7f9b3100d640 10 monclient: _renew_subs 2024-03-25T18:16:31.694 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.692+0000 7f9b3100d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:31.694 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.692+0000 7f9b3100d640 1 -- 172.21.15.204:0/3766373321 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b2c134b10 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.694 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.693+0000 7f9b3100d640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:31.694 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.693+0000 7f9b3100d640 10 monclient: _renew_subs 2024-03-25T18:16:31.694 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.693+0000 7f9b3100d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:31.694 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.693+0000 7f9b3100d640 1 -- 172.21.15.204:0/3766373321 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b2c135320 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.694 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.693+0000 7f9b3100d640 1 librados: init done 2024-03-25T18:16:31.695 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.693+0000 7f9b3100d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:31.695 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.693+0000 7f9b3100d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:31.695 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.693+0000 7f9b3100d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:31.695 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.693+0000 7f9b3100d640 1 -- 172.21.15.204:0/3766373321 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b2c04f730 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.728 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:31.727+0000 7fea77013040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:16:31.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.796+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b1c0047e0 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.796+0000 7f9b137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:31.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.796+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9b1c004af0 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.796+0000 7f9b137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:31.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.796+0000 7f9b137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:31.797 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.796+0000 7f9b137fe640 10 monclient: dump: 2024-03-25T18:16:31.797 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:31.797 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:31.797 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.797 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:31.797 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:31.797 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:31.797 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:31.798 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:31.798 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:31.798 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:31.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.796+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9b1c004fe0 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.798 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.796+0000 7f9b2959a640 4 set_mon_vals no callback set 2024-03-25T18:16:31.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.797+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 922175214 0 0) 0x7f9b1c006570 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.797+0000 7f9b137fe640 10 client.4131.objecter ms_dispatch 0x7f9b2c000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:16:31.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.797+0000 7f9b137fe640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:16:31.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.797+0000 7f9b137fe640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:16:31.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.797+0000 7f9b137fe640 20 client.4131.objecter dump_active .. 0 homeless 2024-03-25T18:16:31.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.797+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9b1c006150 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.797+0000 7f9b137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:31.799 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.797+0000 7f9b137fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:31.832 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:31.832+0000 7f4b797b8040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:16:31.862 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:31.861+0000 7fea77013040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:16:31.868 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.868+0000 7f9b3100d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "188de57d-73fb-49be-84c4-92af3a291860", "id": 2}] 2024-03-25T18:16:31.868 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.868+0000 7f9b3100d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "188de57d-73fb-49be-84c4-92af3a291860", "id": 2}] 2024-03-25T18:16:31.868 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.868+0000 7f9b3100d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:31.868 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.868+0000 7f9b3100d640 1 -- 172.21.15.204:0/3766373321 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd new", "uuid": "188de57d-73fb-49be-84c4-92af3a291860", "id": 2} v 0) v1 -- 0x7f9b2c13e550 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.899 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.899+0000 7f9b137fe640 1 -- 172.21.15.204:0/3766373321 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "188de57d-73fb-49be-84c4-92af3a291860", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1173409666 0 216599125) 0x7f9b1c006860 con 0x7f9b2c0a96b0 2024-03-25T18:16:31.900 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.899+0000 7f9b137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "188de57d-73fb-49be-84c4-92af3a291860", "id": 2}] 2024-03-25T18:16:31.900 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.899+0000 7f9b137fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:31.900 INFO:teuthology.orchestra.run.smithi204.stdout:2 2024-03-25T18:16:31.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.900+0000 7f9b3100d640 10 librados: watch_flush enter 2024-03-25T18:16:31.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.900+0000 7f9b3100d640 10 librados: watch_flush exit 2024-03-25T18:16:31.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.900+0000 7f9b3100d640 20 client.4131.objecter shutdown clearing up homeless session... 2024-03-25T18:16:31.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.900+0000 7f9b3100d640 10 client.4131.objecter successfully canceled tick 2024-03-25T18:16:31.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.900+0000 7f9b3100d640 10 monclient: shutdown 2024-03-25T18:16:31.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.900+0000 7f9b3100d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:31.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.900+0000 7f9b3100d640 1 -- 172.21.15.204:0/3766373321 >> v1:172.21.15.122:6789/0 conn(0x7f9b2c0a96b0 legacy=0x7f9b2c0ac2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:31.901 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.900+0000 7f9b3100d640 1 -- 172.21.15.204:0/3766373321 shutdown_connections 2024-03-25T18:16:31.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.900+0000 7f9b3100d640 1 -- 172.21.15.204:0/3766373321 >> 172.21.15.204:0/3766373321 conn(0x7f9b2c0668e0 msgr2=0x7f9b2c0b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:31.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.900+0000 7f9b3100d640 1 -- 172.21.15.204:0/3766373321 shutdown_connections 2024-03-25T18:16:31.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.900+0000 7f9b3100d640 1 -- 172.21.15.204:0/3766373321 wait complete. 2024-03-25T18:16:31.902 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:31.900+0000 7f9b3100d640 1 librados: shutdown 2024-03-25T18:16:31.912 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new f1f49ae7-e393-4d82-9aa3-153bc6e286d1 3 2024-03-25T18:16:31.974 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:31.974+0000 7f4b797b8040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:16:32.013 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.012+0000 7f9d9b372640 10 monclient: get_monmap_and_config 2024-03-25T18:16:32.013 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.012+0000 7f9d9b372640 10 monclient: build_initial_monmap 2024-03-25T18:16:32.013 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.012+0000 7f9d9b372640 10 monclient: monmap: 2024-03-25T18:16:32.013 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-25T18:16:32.013 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.014 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:16:32.013735+0000 2024-03-25T18:16:32.014 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:16:32.013735+0000 2024-03-25T18:16:32.014 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:32.014 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.014 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:32.014 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:16:32.014 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:16:32.014 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.014 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.013+0000 7f9d9b372640 1 Processor -- start 2024-03-25T18:16:32.015 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.013+0000 7f9d9b372640 1 -- start start 2024-03-25T18:16:32.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.013+0000 7f9d9b372640 10 monclient: init 2024-03-25T18:16:32.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.014+0000 7f9d9b372640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:32.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.014+0000 7f9d9b372640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:32.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.014+0000 7f9d9b372640 10 monclient(hunting): picked mon.noname-b con 0x7f9d940b4700 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:32.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.014+0000 7f9d9b372640 10 monclient(hunting): picked mon.noname-c con 0x7f9d94142a20 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:32.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.014+0000 7f9d9b372640 10 monclient(hunting): picked mon.noname-a con 0x7f9d940a96b0 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:32.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.014+0000 7f9d9b372640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d940a9d60 con 0x7f9d940a96b0 2024-03-25T18:16:32.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.014+0000 7f9d9b372640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9413ff10 con 0x7f9d94142a20 2024-03-25T18:16:32.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.014+0000 7f9d9b372640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d940aa1f0 con 0x7f9d940b4700 2024-03-25T18:16:32.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.014+0000 7f9d9b372640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:32.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.014+0000 7f9d9b372640 10 monclient(hunting): authenticate will time out at 7276.163964s 2024-03-25T18:16:32.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.014+0000 7f9d9990f640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9d940a96b0 0x7f9d9413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.204:37598/0 (socket says 172.21.15.204:37598) 2024-03-25T18:16:32.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.014+0000 7f9d9910e640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f9d940b4700 0x7f9d94142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.204:51318/0 (socket says 172.21.15.204:51318) 2024-03-25T18:16:32.016 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.014+0000 7f9d9990f640 1 -- 172.21.15.204:0/1055022899 learned_addr learned my addr 172.21.15.204:0/1055022899 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-25T18:16:32.017 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.016+0000 7f9d8bfff640 1 -- 172.21.15.204:0/1055022899 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576013369 0 0) 0x7f9d940a9d60 con 0x7f9d940a96b0 2024-03-25T18:16:32.017 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.016+0000 7f9d8bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.017 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.016+0000 7f9d8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.017 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.016+0000 7f9d8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.017 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.016+0000 7f9d8bfff640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:16:32.017 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.016+0000 7f9d8bfff640 1 -- 172.21.15.204:0/1055022899 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d80003520 con 0x7f9d940a96b0 2024-03-25T18:16:32.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.017+0000 7f9d8bfff640 1 -- 172.21.15.204:0/1055022899 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144698556 0 0) 0x7f9d940aa1f0 con 0x7f9d940b4700 2024-03-25T18:16:32.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.017+0000 7f9d8bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.017+0000 7f9d8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.017+0000 7f9d8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.017+0000 7f9d8bfff640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:16:32.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.017+0000 7f9d8bfff640 1 -- 172.21.15.204:0/1055022899 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d940a9d60 con 0x7f9d940b4700 2024-03-25T18:16:32.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.017+0000 7f9d8bfff640 1 -- 172.21.15.204:0/1055022899 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1408009903 0 0) 0x7f9d9413ff10 con 0x7f9d94142a20 2024-03-25T18:16:32.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.017+0000 7f9d8bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.018 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.017+0000 7f9d8bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.019 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.017+0000 7f9d8bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.019 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.017+0000 7f9d8bfff640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:16:32.019 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.017+0000 7f9d8bfff640 1 -- 172.21.15.204:0/1055022899 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d940aa1f0 con 0x7f9d94142a20 2024-03-25T18:16:32.021 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.020+0000 7f9d8bfff640 1 -- 172.21.15.204:0/1055022899 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2741417919 0 0) 0x7f9d80003520 con 0x7f9d940a96b0 2024-03-25T18:16:32.021 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.020+0000 7f9d8bfff640 1 -- 172.21.15.204:0/1055022899 >> v1:172.21.15.122:6790/0 conn(0x7f9d940b4700 legacy=0x7f9d94142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.021 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.020+0000 7f9d8bfff640 1 -- 172.21.15.204:0/1055022899 >> v1:172.21.15.204:6789/0 conn(0x7f9d94142a20 legacy=0x7f9d9413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.021 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.020+0000 7f9d8bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:32.021 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.020+0000 7f9d8bfff640 1 monclient: found mon.noname-a 2024-03-25T18:16:32.021 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.020+0000 7f9d8bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:32.021 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.020+0000 7f9d8bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:16:32.021 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.020+0000 7f9d8bfff640 1 -- 172.21.15.204:0/1055022899 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d940b4ae0 con 0x7f9d940a96b0 2024-03-25T18:16:32.021 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.020+0000 7f9d8bfff640 10 monclient: _finish_auth 0 2024-03-25T18:16:32.021 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.020+0000 7f9d8bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:32.021 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.020+0000 7f9d8bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:32.022 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.020+0000 7f9d8bfff640 1 -- 172.21.15.204:0/1055022899 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9d8c001a10 con 0x7f9d940a96b0 2024-03-25T18:16:32.022 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.020+0000 7f9d8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.022 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.020+0000 7f9d8bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:32.022 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.020+0000 7f9d8bfff640 10 monclient: dump: 2024-03-25T18:16:32.022 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.022 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.022 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.022 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.022 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.022 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.022 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.022 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.022 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.022 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.023 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.020+0000 7f9d9b372640 5 monclient: authenticate success, global_id 4134 2024-03-25T18:16:32.023 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.020+0000 7f9d9b372640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:32.025 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.021+0000 7f9d8bfff640 1 -- 172.21.15.204:0/1055022899 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d8c001b70 con 0x7f9d940a96b0 2024-03-25T18:16:32.025 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.021+0000 7f9d8bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:32.026 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.021+0000 7f9d8bfff640 1 -- 172.21.15.204:0/1055022899 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9d8c001f80 con 0x7f9d940a96b0 2024-03-25T18:16:32.026 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.021+0000 7f9d9b372640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:32.026 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.021+0000 7f9d9b372640 4 set_mon_vals no callback set 2024-03-25T18:16:32.026 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.021+0000 7f9d8bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.026 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.021+0000 7f9d8bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:32.026 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.021+0000 7f9d8bfff640 10 monclient: dump: 2024-03-25T18:16:32.026 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.026 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.026 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.026 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.026 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.026 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.026 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.026 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.027 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.027 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.027 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.022+0000 7f9d9b372640 10 monclient: shutdown 2024-03-25T18:16:32.027 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.022+0000 7f9d9b372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:32.027 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.022+0000 7f9d9b372640 1 -- 172.21.15.204:0/1055022899 >> v1:172.21.15.122:6789/0 conn(0x7f9d940a96b0 legacy=0x7f9d9413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.027 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.022+0000 7f9d9b372640 1 -- 172.21.15.204:0/1055022899 shutdown_connections 2024-03-25T18:16:32.027 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.022+0000 7f9d9b372640 1 -- 172.21.15.204:0/1055022899 >> 172.21.15.204:0/1055022899 conn(0x7f9d940668e0 msgr2=0x7f9d94066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:32.027 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.022+0000 7f9d9b372640 1 -- 172.21.15.204:0/1055022899 shutdown_connections 2024-03-25T18:16:32.027 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.022+0000 7f9d9b372640 1 -- 172.21.15.204:0/1055022899 wait complete. 2024-03-25T18:16:32.027 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.023+0000 7f9d9b372640 10 monclient: build_initial_monmap 2024-03-25T18:16:32.027 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.023+0000 7f9d9b372640 10 monclient: monmap: 2024-03-25T18:16:32.027 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-25T18:16:32.027 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:32.027 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-03-25T18:16:32.027 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-03-25T18:16:32.028 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:32.028 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-25T18:16:32.028 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:32.028 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:16:32.028 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:16:32.028 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.028 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.023+0000 7f9d9b372640 1 librados: starting msgr at 2024-03-25T18:16:32.028 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.023+0000 7f9d9b372640 1 librados: starting objecter 2024-03-25T18:16:32.028 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.023+0000 7f9d9b372640 1 Processor -- start 2024-03-25T18:16:32.028 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.023+0000 7f9d9b372640 1 -- start start 2024-03-25T18:16:32.028 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.023+0000 7f9d9b372640 1 librados: setting wanted keys 2024-03-25T18:16:32.028 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.023+0000 7f9d9b372640 1 librados: calling monclient init 2024-03-25T18:16:32.028 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.023+0000 7f9d9b372640 10 monclient: init 2024-03-25T18:16:32.028 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.024+0000 7f9d9b372640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:32.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.024+0000 7f9d9b372640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:32.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.024+0000 7f9d9b372640 10 monclient(hunting): picked mon.noname-a con 0x7f9d940a96b0 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:32.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.024+0000 7f9d9b372640 10 monclient(hunting): picked mon.noname-c con 0x7f9d940b4700 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:32.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.024+0000 7f9d9b372640 10 monclient(hunting): picked mon.noname-b con 0x7f9d94142a20 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:32.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.024+0000 7f9d9b372640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d941db820 con 0x7f9d940a96b0 2024-03-25T18:16:32.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.024+0000 7f9d9b372640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d9413cd70 con 0x7f9d94142a20 2024-03-25T18:16:32.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.024+0000 7f9d9b372640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d940b1af0 con 0x7f9d940b4700 2024-03-25T18:16:32.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.024+0000 7f9d9b372640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:32.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.024+0000 7f9d9b372640 10 monclient(hunting): authenticate will time out at 7276.174010s 2024-03-25T18:16:32.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.024+0000 7f9d9910e640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9d940a96b0 0x7f9d941dab60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.204:37606/0 (socket says 172.21.15.204:37606) 2024-03-25T18:16:32.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.024+0000 7f9d9910e640 1 -- 172.21.15.204:0/3246786752 learned_addr learned my addr 172.21.15.204:0/3246786752 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-25T18:16:32.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.025+0000 7f9d89ffb640 10 client.?.objecter ms_handle_connect 0x7f9d940b4700 2024-03-25T18:16:32.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.025+0000 7f9d89ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.025+0000 7f9d89ffb640 10 client.?.objecter ms_handle_connect 0x7f9d940a96b0 2024-03-25T18:16:32.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.025+0000 7f9d89ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.029 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.025+0000 7f9d89ffb640 10 client.?.objecter ms_handle_connect 0x7f9d94142a20 2024-03-25T18:16:32.030 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.025+0000 7f9d89ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 248427677 0 0) 0x7f9d941db820 con 0x7f9d940a96b0 2024-03-25T18:16:32.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:16:32.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d78004eb0 con 0x7f9d940a96b0 2024-03-25T18:16:32.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 125755203 0 0) 0x7f9d940b1af0 con 0x7f9d940b4700 2024-03-25T18:16:32.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:16:32.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d941db820 con 0x7f9d940b4700 2024-03-25T18:16:32.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1904726276 0 0) 0x7f9d9413cd70 con 0x7f9d94142a20 2024-03-25T18:16:32.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.031 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:16:32.032 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.027+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d940b1af0 con 0x7f9d94142a20 2024-03-25T18:16:32.033 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.031+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 748101724 0 0) 0x7f9d78004eb0 con 0x7f9d940a96b0 2024-03-25T18:16:32.033 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.031+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d9413cd70 con 0x7f9d940a96b0 2024-03-25T18:16:32.033 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.031+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9d900019f0 con 0x7f9d940a96b0 2024-03-25T18:16:32.033 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.031+0000 7f9d89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.033 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.031+0000 7f9d89ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:32.033 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.031+0000 7f9d89ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:32.034 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.034 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.034 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.034 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.034 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.034 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.034 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.034 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.034 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.034 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.034 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.031+0000 7f9d89ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:32.034 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.031+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1435861394 0 0) 0x7f9d941db820 con 0x7f9d940b4700 2024-03-25T18:16:32.034 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.031+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d78004eb0 con 0x7f9d940b4700 2024-03-25T18:16:32.034 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.031+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9d84001870 con 0x7f9d940b4700 2024-03-25T18:16:32.034 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.031+0000 7f9d89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.035 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.031+0000 7f9d89ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:32.035 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.031+0000 7f9d89ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:32.035 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.035 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.035 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.035 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.035 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.035 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.035 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.035 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.035 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.035 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.035 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.032+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2679129925 0 0) 0x7f9d940b1af0 con 0x7f9d94142a20 2024-03-25T18:16:32.035 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.032+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9d941db820 con 0x7f9d94142a20 2024-03-25T18:16:32.036 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.032+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9d8c001e80 con 0x7f9d94142a20 2024-03-25T18:16:32.036 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.032+0000 7f9d89ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.036 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.032+0000 7f9d89ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:32.036 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.032+0000 7f9d89ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:32.036 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.036 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.036 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.036 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.036 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.036 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.036 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.036 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.036 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.036 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.038 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.034+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3768773195 0 0) 0x7f9d9413cd70 con 0x7f9d940a96b0 2024-03-25T18:16:32.038 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.034+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 >> v1:172.21.15.122:6790/0 conn(0x7f9d940b4700 legacy=0x7f9d941d42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.038 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.034+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 >> v1:172.21.15.204:6789/0 conn(0x7f9d94142a20 legacy=0x7f9d941d49c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.038 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.034+0000 7f9d89ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:32.038 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.034+0000 7f9d89ffb640 1 monclient: found mon.a 2024-03-25T18:16:32.038 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.034+0000 7f9d89ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:32.038 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.034+0000 7f9d89ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:32.038 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.034+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d940b1c80 con 0x7f9d940a96b0 2024-03-25T18:16:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.034+0000 7f9d89ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.034+0000 7f9d89ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.034+0000 7f9d89ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.034+0000 7f9d9b372640 5 monclient: authenticate success, global_id 4137 2024-03-25T18:16:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.034+0000 7f9d9b372640 10 monclient: _renew_subs 2024-03-25T18:16:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.034+0000 7f9d9b372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.034+0000 7f9d9b372640 1 -- 172.21.15.204:0/3246786752 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d941db270 con 0x7f9d940a96b0 2024-03-25T18:16:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.035+0000 7f9d9b372640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.035+0000 7f9d9b372640 10 monclient: _renew_subs 2024-03-25T18:16:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.035+0000 7f9d9b372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.035+0000 7f9d9b372640 1 -- 172.21.15.204:0/3246786752 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d941db560 con 0x7f9d940a96b0 2024-03-25T18:16:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.035+0000 7f9d9b372640 1 librados: init done 2024-03-25T18:16:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.035+0000 7f9d9b372640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:32.039 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.035+0000 7f9d9b372640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:32.040 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.036+0000 7f9d9b372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:32.040 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.036+0000 7f9d9b372640 1 -- 172.21.15.204:0/3246786752 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d9404f730 con 0x7f9d940a96b0 2024-03-25T18:16:32.109 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:32.109+0000 7f4b797b8040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:16:32.128 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.127+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9d900047e0 con 0x7f9d940a96b0 2024-03-25T18:16:32.128 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.127+0000 7f9d89ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:32.128 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.127+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9d90004af0 con 0x7f9d940a96b0 2024-03-25T18:16:32.128 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.127+0000 7f9d89ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.128 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.127+0000 7f9d89ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:32.128 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.127+0000 7f9d89ffb640 10 monclient: dump: 2024-03-25T18:16:32.128 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.129 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.129 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.129 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.129 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.129 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.129 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.129 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.129 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.129 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.129 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.127+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9d90005060 con 0x7f9d940a96b0 2024-03-25T18:16:32.129 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.127+0000 7f9d8b7fe640 4 set_mon_vals no callback set 2024-03-25T18:16:32.130 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.128+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 1418089852 0 0) 0x7f9d900066b0 con 0x7f9d940a96b0 2024-03-25T18:16:32.130 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.128+0000 7f9d89ffb640 10 client.4137.objecter ms_dispatch 0x7f9d94000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:16:32.130 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.128+0000 7f9d89ffb640 3 client.4137.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:16:32.130 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.128+0000 7f9d89ffb640 3 client.4137.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:16:32.130 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.128+0000 7f9d89ffb640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-25T18:16:32.130 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.128+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9d90006220 con 0x7f9d940a96b0 2024-03-25T18:16:32.130 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.128+0000 7f9d89ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:32.130 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.128+0000 7f9d89ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:32.157 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:32.156+0000 7f4b797b8040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:16:32.201 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.201+0000 7f9d9b372640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f1f49ae7-e393-4d82-9aa3-153bc6e286d1", "id": 3}] 2024-03-25T18:16:32.201 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.201+0000 7f9d9b372640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f1f49ae7-e393-4d82-9aa3-153bc6e286d1", "id": 3}] 2024-03-25T18:16:32.201 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.201+0000 7f9d9b372640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:32.201 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.201+0000 7f9d9b372640 1 -- 172.21.15.204:0/3246786752 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f1f49ae7-e393-4d82-9aa3-153bc6e286d1", "id": 3} v 0) v1 -- 0x7f9d94069f30 con 0x7f9d940a96b0 2024-03-25T18:16:32.254 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.254+0000 7f9d89ffb640 1 -- 172.21.15.204:0/3246786752 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f1f49ae7-e393-4d82-9aa3-153bc6e286d1", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 717516427 0 525046306) 0x7f9d90006a20 con 0x7f9d940a96b0 2024-03-25T18:16:32.255 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.254+0000 7f9d89ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f1f49ae7-e393-4d82-9aa3-153bc6e286d1", "id": 3}] 2024-03-25T18:16:32.255 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.254+0000 7f9d89ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:32.255 INFO:teuthology.orchestra.run.smithi204.stdout:3 2024-03-25T18:16:32.256 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.255+0000 7f9d9b372640 10 librados: watch_flush enter 2024-03-25T18:16:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.255+0000 7f9d9b372640 10 librados: watch_flush exit 2024-03-25T18:16:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.255+0000 7f9d9b372640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-25T18:16:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.255+0000 7f9d9b372640 10 client.4137.objecter successfully canceled tick 2024-03-25T18:16:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.255+0000 7f9d9b372640 10 monclient: shutdown 2024-03-25T18:16:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.255+0000 7f9d9b372640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.255+0000 7f9d9b372640 1 -- 172.21.15.204:0/3246786752 >> v1:172.21.15.122:6789/0 conn(0x7f9d940a96b0 legacy=0x7f9d941dab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.255+0000 7f9d9b372640 1 -- 172.21.15.204:0/3246786752 shutdown_connections 2024-03-25T18:16:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.255+0000 7f9d9b372640 1 -- 172.21.15.204:0/3246786752 >> 172.21.15.204:0/3246786752 conn(0x7f9d940668e0 msgr2=0x7f9d940b5e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.255+0000 7f9d9b372640 1 -- 172.21.15.204:0/3246786752 shutdown_connections 2024-03-25T18:16:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.255+0000 7f9d9b372640 1 -- 172.21.15.204:0/3246786752 wait complete. 2024-03-25T18:16:32.257 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.255+0000 7f9d9b372640 1 librados: shutdown 2024-03-25T18:16:32.266 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new eba478f5-2079-4357-8b03-5604ac7945c0 4 2024-03-25T18:16:32.267 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:32.266+0000 7fea77013040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:16:32.275 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:32.275+0000 7f4b797b8040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:16:32.311 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:32.312+0000 7fea77013040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:16:32.367 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.366+0000 7f0e807cf640 10 monclient: get_monmap_and_config 2024-03-25T18:16:32.368 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.366+0000 7f0e807cf640 10 monclient: build_initial_monmap 2024-03-25T18:16:32.368 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.366+0000 7f0e807cf640 10 monclient: monmap: 2024-03-25T18:16:32.368 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-25T18:16:32.368 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.368 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:16:32.367745+0000 2024-03-25T18:16:32.368 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:16:32.367745+0000 2024-03-25T18:16:32.368 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:32.368 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.368 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:32.368 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:16:32.368 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:16:32.368 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.368 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.367+0000 7f0e807cf640 1 Processor -- start 2024-03-25T18:16:32.368 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.367+0000 7f0e807cf640 1 -- start start 2024-03-25T18:16:32.369 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.367+0000 7f0e807cf640 10 monclient: init 2024-03-25T18:16:32.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.368+0000 7f0e807cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:32.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.368+0000 7f0e807cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:32.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.368+0000 7f0e807cf640 10 monclient(hunting): picked mon.noname-c con 0x7f0e780b4700 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:32.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.368+0000 7f0e807cf640 10 monclient(hunting): picked mon.noname-b con 0x7f0e78142a20 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:32.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.368+0000 7f0e807cf640 10 monclient(hunting): picked mon.noname-a con 0x7f0e780a96b0 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:32.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.368+0000 7f0e807cf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e780a9d60 con 0x7f0e780a96b0 2024-03-25T18:16:32.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.368+0000 7f0e807cf640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7813ff10 con 0x7f0e780b4700 2024-03-25T18:16:32.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.368+0000 7f0e807cf640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e780aa1f0 con 0x7f0e78142a20 2024-03-25T18:16:32.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.368+0000 7f0e807cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:32.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.368+0000 7f0e807cf640 10 monclient(hunting): authenticate will time out at 7276.517927s 2024-03-25T18:16:32.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.368+0000 7f0e7ed6c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0e780a96b0 0x7f0e7813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.204:37614/0 (socket says 172.21.15.204:37614) 2024-03-25T18:16:32.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.368+0000 7f0e7dd6a640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f0e78142a20 0x7f0e7813f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.204:51322/0 (socket says 172.21.15.204:51322) 2024-03-25T18:16:32.370 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.368+0000 7f0e7ed6c640 1 -- 172.21.15.204:0/3907075676 learned_addr learned my addr 172.21.15.204:0/3907075676 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-25T18:16:32.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 1 -- 172.21.15.204:0/3907075676 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716905287 0 0) 0x7f0e780a9d60 con 0x7f0e780a96b0 2024-03-25T18:16:32.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.371 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:16:32.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 1 -- 172.21.15.204:0/3907075676 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e64003520 con 0x7f0e780a96b0 2024-03-25T18:16:32.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 1 -- 172.21.15.204:0/3907075676 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042930549 0 0) 0x7f0e780aa1f0 con 0x7f0e78142a20 2024-03-25T18:16:32.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:16:32.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 1 -- 172.21.15.204:0/3907075676 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e780a9d60 con 0x7f0e78142a20 2024-03-25T18:16:32.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 1 -- 172.21.15.204:0/3907075676 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1843543334 0 0) 0x7f0e7813ff10 con 0x7f0e780b4700 2024-03-25T18:16:32.372 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.373 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.373 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.373 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:16:32.373 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.371+0000 7f0e7d569640 1 -- 172.21.15.204:0/3907075676 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e780aa1f0 con 0x7f0e780b4700 2024-03-25T18:16:32.375 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e7d569640 1 -- 172.21.15.204:0/3907075676 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2835165686 0 0) 0x7f0e64003520 con 0x7f0e780a96b0 2024-03-25T18:16:32.375 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e7d569640 1 -- 172.21.15.204:0/3907075676 >> v1:172.21.15.122:6790/0 conn(0x7f0e78142a20 legacy=0x7f0e7813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.375 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e7d569640 1 -- 172.21.15.204:0/3907075676 >> v1:172.21.15.204:6789/0 conn(0x7f0e780b4700 legacy=0x7f0e78142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.375 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e7d569640 10 monclient: _finish_hunting 0 2024-03-25T18:16:32.375 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e7d569640 1 monclient: found mon.noname-a 2024-03-25T18:16:32.375 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e7d569640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:32.375 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e7d569640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:16:32.375 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e7d569640 1 -- 172.21.15.204:0/3907075676 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e780b4ae0 con 0x7f0e780a96b0 2024-03-25T18:16:32.375 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e7d569640 10 monclient: _finish_auth 0 2024-03-25T18:16:32.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e7d569640 10 monclient: _check_auth_tickets 2024-03-25T18:16:32.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e7d569640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:32.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e7d569640 1 -- 172.21.15.204:0/3907075676 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0e70001a10 con 0x7f0e780a96b0 2024-03-25T18:16:32.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e807cf640 5 monclient: authenticate success, global_id 4140 2024-03-25T18:16:32.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e807cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:32.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e7d569640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e7d569640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:32.376 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e7d569640 10 monclient: dump: 2024-03-25T18:16:32.376 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.376 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.376 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.376 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.376 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.376 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.376 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.376 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.377 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.377 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.377 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.374+0000 7f0e807cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:32.379 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.376+0000 7f0e7d569640 1 -- 172.21.15.204:0/3907075676 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e70001b70 con 0x7f0e780a96b0 2024-03-25T18:16:32.379 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.376+0000 7f0e7d569640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:32.379 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.376+0000 7f0e807cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:32.379 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.376+0000 7f0e7d569640 1 -- 172.21.15.204:0/3907075676 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0e70001fc0 con 0x7f0e780a96b0 2024-03-25T18:16:32.380 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.376+0000 7f0e807cf640 4 set_mon_vals no callback set 2024-03-25T18:16:32.380 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.376+0000 7f0e7d569640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.380 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.376+0000 7f0e7d569640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:32.380 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.376+0000 7f0e7d569640 10 monclient: dump: 2024-03-25T18:16:32.380 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.380 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.380 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.380 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.380 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.380 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.380 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.380 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.380 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.380 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.380 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.376+0000 7f0e807cf640 10 monclient: shutdown 2024-03-25T18:16:32.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.376+0000 7f0e807cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:32.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.376+0000 7f0e807cf640 1 -- 172.21.15.204:0/3907075676 >> v1:172.21.15.122:6789/0 conn(0x7f0e780a96b0 legacy=0x7f0e7813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.376+0000 7f0e807cf640 1 -- 172.21.15.204:0/3907075676 shutdown_connections 2024-03-25T18:16:32.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.376+0000 7f0e807cf640 1 -- 172.21.15.204:0/3907075676 >> 172.21.15.204:0/3907075676 conn(0x7f0e780668e0 msgr2=0x7f0e78066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:32.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.376+0000 7f0e807cf640 1 -- 172.21.15.204:0/3907075676 shutdown_connections 2024-03-25T18:16:32.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.376+0000 7f0e807cf640 1 -- 172.21.15.204:0/3907075676 wait complete. 2024-03-25T18:16:32.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.377+0000 7f0e807cf640 10 monclient: build_initial_monmap 2024-03-25T18:16:32.381 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.377+0000 7f0e807cf640 10 monclient: monmap: 2024-03-25T18:16:32.381 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-25T18:16:32.381 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:32.381 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-03-25T18:16:32.381 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-03-25T18:16:32.381 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:32.381 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-25T18:16:32.381 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:32.381 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:16:32.382 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:16:32.382 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.377+0000 7f0e807cf640 1 librados: starting msgr at 2024-03-25T18:16:32.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.377+0000 7f0e807cf640 1 librados: starting objecter 2024-03-25T18:16:32.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.377+0000 7f0e807cf640 1 Processor -- start 2024-03-25T18:16:32.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.377+0000 7f0e807cf640 1 -- start start 2024-03-25T18:16:32.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.377+0000 7f0e807cf640 1 librados: setting wanted keys 2024-03-25T18:16:32.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.377+0000 7f0e807cf640 1 librados: calling monclient init 2024-03-25T18:16:32.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.377+0000 7f0e807cf640 10 monclient: init 2024-03-25T18:16:32.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.378+0000 7f0e807cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:32.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.378+0000 7f0e807cf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:32.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.378+0000 7f0e807cf640 10 monclient(hunting): picked mon.noname-b con 0x7f0e780a96b0 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:32.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.378+0000 7f0e807cf640 10 monclient(hunting): picked mon.noname-a con 0x7f0e780b4700 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:32.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.378+0000 7f0e807cf640 10 monclient(hunting): picked mon.noname-c con 0x7f0e78142a20 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:32.382 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.378+0000 7f0e807cf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7813c800 con 0x7f0e780b4700 2024-03-25T18:16:32.383 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.378+0000 7f0e807cf640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7813cbd0 con 0x7f0e780a96b0 2024-03-25T18:16:32.383 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.378+0000 7f0e807cf640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0e7813cd60 con 0x7f0e78142a20 2024-03-25T18:16:32.383 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.378+0000 7f0e807cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:32.383 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.378+0000 7f0e807cf640 10 monclient(hunting): authenticate will time out at 7276.528211s 2024-03-25T18:16:32.383 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.378+0000 7f0e7dd6a640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0e780b4700 0x7f0e780ac930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.204:37616/0 (socket says 172.21.15.204:37616) 2024-03-25T18:16:32.383 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.378+0000 7f0e7dd6a640 1 -- 172.21.15.204:0/4256175074 learned_addr learned my addr 172.21.15.204:0/4256175074 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-25T18:16:32.383 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:32.379+0000 7f4b797b8040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:16:32.384 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.379+0000 7f0e6effd640 10 client.?.objecter ms_handle_connect 0x7f0e780b4700 2024-03-25T18:16:32.384 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.379+0000 7f0e6effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.384 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.379+0000 7f0e6effd640 10 client.?.objecter ms_handle_connect 0x7f0e78142a20 2024-03-25T18:16:32.384 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.379+0000 7f0e6effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.384 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.379+0000 7f0e6effd640 10 client.?.objecter ms_handle_connect 0x7f0e780a96b0 2024-03-25T18:16:32.385 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.379+0000 7f0e6effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.385 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855118305 0 0) 0x7f0e7813c800 con 0x7f0e780b4700 2024-03-25T18:16:32.385 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.385 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.385 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.385 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:16:32.385 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e5c004eb0 con 0x7f0e780b4700 2024-03-25T18:16:32.385 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1926991226 0 0) 0x7f0e7813cd60 con 0x7f0e78142a20 2024-03-25T18:16:32.385 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.385 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.385 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.385 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:16:32.385 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e7813c800 con 0x7f0e78142a20 2024-03-25T18:16:32.385 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214718543 0 0) 0x7f0e7813cbd0 con 0x7f0e780a96b0 2024-03-25T18:16:32.385 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.386 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.386 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.386 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:16:32.386 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.381+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e7813cd60 con 0x7f0e780a96b0 2024-03-25T18:16:32.387 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.385+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3061478260 0 0) 0x7f0e5c004eb0 con 0x7f0e780b4700 2024-03-25T18:16:32.387 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.385+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e7813cbd0 con 0x7f0e780b4700 2024-03-25T18:16:32.387 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.385+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0e74001a50 con 0x7f0e780b4700 2024-03-25T18:16:32.387 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.385+0000 7f0e6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.387 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.385+0000 7f0e6effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:32.387 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.385+0000 7f0e6effd640 10 monclient(hunting): dump: 2024-03-25T18:16:32.388 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.388 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.388 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.388 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.388 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.388 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.388 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.388 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.388 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.388 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.388 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.385+0000 7f0e6effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:32.388 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.385+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 786925590 0 0) 0x7f0e7813c800 con 0x7f0e78142a20 2024-03-25T18:16:32.388 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.385+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e5c004eb0 con 0x7f0e78142a20 2024-03-25T18:16:32.388 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.385+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0e70001e80 con 0x7f0e78142a20 2024-03-25T18:16:32.389 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.385+0000 7f0e6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.389 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.386+0000 7f0e6effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:32.389 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.386+0000 7f0e6effd640 10 monclient(hunting): dump: 2024-03-25T18:16:32.389 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.389 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.389 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.389 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.389 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.389 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.389 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.389 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.389 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.389 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.389 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.386+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3016411435 0 0) 0x7f0e7813cd60 con 0x7f0e780a96b0 2024-03-25T18:16:32.389 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.386+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e7813c800 con 0x7f0e780a96b0 2024-03-25T18:16:32.389 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.386+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0e68001830 con 0x7f0e780a96b0 2024-03-25T18:16:32.390 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.386+0000 7f0e6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.390 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.386+0000 7f0e6effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:32.390 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.386+0000 7f0e6effd640 10 monclient(hunting): dump: 2024-03-25T18:16:32.390 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.390 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.390 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.390 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.390 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.390 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.390 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.390 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.390 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.390 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.392 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3251726537 0 0) 0x7f0e7813cbd0 con 0x7f0e780b4700 2024-03-25T18:16:32.392 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 >> v1:172.21.15.122:6790/0 conn(0x7f0e78142a20 legacy=0x7f0e781e2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.392 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 >> v1:172.21.15.204:6789/0 conn(0x7f0e780a96b0 legacy=0x7f0e7813c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.392 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e6effd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:32.392 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e6effd640 1 monclient: found mon.a 2024-03-25T18:16:32.392 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:32.392 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e6effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:32.392 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e78139fb0 con 0x7f0e780b4700 2024-03-25T18:16:32.392 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e6effd640 10 monclient: _finish_auth 0 2024-03-25T18:16:32.392 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e6effd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:32.392 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:32.392 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e807cf640 5 monclient: authenticate success, global_id 4143 2024-03-25T18:16:32.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e807cf640 10 monclient: _renew_subs 2024-03-25T18:16:32.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e807cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:32.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e807cf640 1 -- 172.21.15.204:0/4256175074 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e780ad040 con 0x7f0e780b4700 2024-03-25T18:16:32.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e807cf640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:32.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e807cf640 10 monclient: _renew_subs 2024-03-25T18:16:32.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e807cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:32.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e807cf640 1 -- 172.21.15.204:0/4256175074 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7813bb10 con 0x7f0e780b4700 2024-03-25T18:16:32.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.389+0000 7f0e807cf640 1 librados: init done 2024-03-25T18:16:32.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.390+0000 7f0e807cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:32.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.390+0000 7f0e807cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:32.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.390+0000 7f0e807cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:32.393 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.390+0000 7f0e807cf640 1 -- 172.21.15.204:0/4256175074 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e7813a200 con 0x7f0e780b4700 2024-03-25T18:16:32.394 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.393+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e74001c50 con 0x7f0e780b4700 2024-03-25T18:16:32.394 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.393+0000 7f0e6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:32.394 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.393+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0e74004c30 con 0x7f0e780b4700 2024-03-25T18:16:32.394 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.393+0000 7f0e6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.394 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.393+0000 7f0e6effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:32.394 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.393+0000 7f0e6effd640 10 monclient: dump: 2024-03-25T18:16:32.394 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.394 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.395 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.395 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.395 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.395 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.395 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.395 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.395 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.395 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.395 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.393+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f0e74005160 con 0x7f0e780b4700 2024-03-25T18:16:32.395 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.393+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1688484792 0 0) 0x7f0e74006830 con 0x7f0e780b4700 2024-03-25T18:16:32.395 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.393+0000 7f0e6effd640 10 client.4143.objecter ms_dispatch 0x7f0e78000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:16:32.395 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.393+0000 7f0e6effd640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:16:32.395 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.393+0000 7f0e7cd68640 4 set_mon_vals no callback set 2024-03-25T18:16:32.395 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.393+0000 7f0e6effd640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:16:32.395 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.393+0000 7f0e6effd640 20 client.4143.objecter dump_active .. 0 homeless 2024-03-25T18:16:32.427 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:32.427+0000 7fea77013040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:16:32.486 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.485+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f0e74006430 con 0x7f0e780b4700 2024-03-25T18:16:32.486 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.485+0000 7f0e6effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:32.486 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.485+0000 7f0e6effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:32.521 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:32.521+0000 7fea77013040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:16:32.538 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:32.538+0000 7f4b797b8040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:16:32.557 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.556+0000 7f0e807cf640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "eba478f5-2079-4357-8b03-5604ac7945c0", "id": 4}] 2024-03-25T18:16:32.557 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.556+0000 7f0e807cf640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "eba478f5-2079-4357-8b03-5604ac7945c0", "id": 4}] 2024-03-25T18:16:32.557 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.556+0000 7f0e807cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:32.557 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.556+0000 7f0e807cf640 1 -- 172.21.15.204:0/4256175074 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd new", "uuid": "eba478f5-2079-4357-8b03-5604ac7945c0", "id": 4} v 0) v1 -- 0x7f0e78139be0 con 0x7f0e780b4700 2024-03-25T18:16:32.582 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.581+0000 7f0e6effd640 1 -- 172.21.15.204:0/4256175074 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "eba478f5-2079-4357-8b03-5604ac7945c0", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 4140898514 0 1697012583) 0x7f0e74006a40 con 0x7f0e780b4700 2024-03-25T18:16:32.582 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.581+0000 7f0e6effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "eba478f5-2079-4357-8b03-5604ac7945c0", "id": 4}] 2024-03-25T18:16:32.582 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.581+0000 7f0e6effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:32.582 INFO:teuthology.orchestra.run.smithi204.stdout:4 2024-03-25T18:16:32.583 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.582+0000 7f0e807cf640 10 librados: watch_flush enter 2024-03-25T18:16:32.583 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.582+0000 7f0e807cf640 10 librados: watch_flush exit 2024-03-25T18:16:32.584 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.582+0000 7f0e807cf640 20 client.4143.objecter shutdown clearing up homeless session... 2024-03-25T18:16:32.584 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.582+0000 7f0e807cf640 10 client.4143.objecter successfully canceled tick 2024-03-25T18:16:32.584 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.582+0000 7f0e807cf640 10 monclient: shutdown 2024-03-25T18:16:32.584 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.582+0000 7f0e807cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:32.584 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.582+0000 7f0e807cf640 1 -- 172.21.15.204:0/4256175074 >> v1:172.21.15.122:6789/0 conn(0x7f0e780b4700 legacy=0x7f0e780ac930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.584 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.582+0000 7f0e807cf640 1 -- 172.21.15.204:0/4256175074 shutdown_connections 2024-03-25T18:16:32.584 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.582+0000 7f0e807cf640 1 -- 172.21.15.204:0/4256175074 >> 172.21.15.204:0/4256175074 conn(0x7f0e780668e0 msgr2=0x7f0e780b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:32.584 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.582+0000 7f0e807cf640 1 -- 172.21.15.204:0/4256175074 shutdown_connections 2024-03-25T18:16:32.584 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.582+0000 7f0e807cf640 1 -- 172.21.15.204:0/4256175074 wait complete. 2024-03-25T18:16:32.584 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.582+0000 7f0e807cf640 1 librados: shutdown 2024-03-25T18:16:32.584 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:32.582+0000 7f4b797b8040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:16:32.593 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new f8aa37f0-78b7-4515-952e-7e25f02c1e3b 5 2024-03-25T18:16:32.684 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:32.684+0000 7fea77013040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:16:32.726 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:32.726+0000 7fea77013040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:16:32.738 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.735+0000 7f3c5b55d640 10 monclient: get_monmap_and_config 2024-03-25T18:16:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.735+0000 7f3c5b55d640 10 monclient: build_initial_monmap 2024-03-25T18:16:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.735+0000 7f3c5b55d640 10 monclient: monmap: 2024-03-25T18:16:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-25T18:16:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:16:32.736427+0000 2024-03-25T18:16:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:16:32.736427+0000 2024-03-25T18:16:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:16:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:16:32.739 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.739 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.736+0000 7f3c5b55d640 1 Processor -- start 2024-03-25T18:16:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.736+0000 7f3c5b55d640 1 -- start start 2024-03-25T18:16:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.736+0000 7f3c5b55d640 10 monclient: init 2024-03-25T18:16:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.736+0000 7f3c5b55d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.736+0000 7f3c5b55d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.736+0000 7f3c5b55d640 10 monclient(hunting): picked mon.noname-b con 0x7f3c5406c2b0 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.736+0000 7f3c5b55d640 10 monclient(hunting): picked mon.noname-c con 0x7f3c540b0100 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.736+0000 7f3c5b55d640 10 monclient(hunting): picked mon.noname-a con 0x7f3c54146300 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.736+0000 7f3c5b55d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5413feb0 con 0x7f3c54146300 2024-03-25T18:16:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.736+0000 7f3c5b55d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5406d050 con 0x7f3c540b0100 2024-03-25T18:16:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.736+0000 7f3c5b55d640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c540b06d0 con 0x7f3c5406c2b0 2024-03-25T18:16:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.736+0000 7f3c5b55d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.736+0000 7f3c5b55d640 10 monclient(hunting): authenticate will time out at 7276.886651s 2024-03-25T18:16:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.737+0000 7f3c592f9640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f3c5406c2b0 0x7f3c540af9f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.204:51334/0 (socket says 172.21.15.204:51334) 2024-03-25T18:16:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.737+0000 7f3c58af8640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f3c540b0100 0x7f3c54145bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.204:33776/0 (socket says 172.21.15.204:33776) 2024-03-25T18:16:32.740 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.737+0000 7f3c59afa640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f3c54146300 0x7f3c5414db40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.204:37622/0 (socket says 172.21.15.204:37622) 2024-03-25T18:16:32.741 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.737+0000 7f3c592f9640 1 -- 172.21.15.204:0/1279543061 learned_addr learned my addr 172.21.15.204:0/1279543061 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-25T18:16:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.739+0000 7f3c4bfff640 1 -- 172.21.15.204:0/1279543061 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379517453 0 0) 0x7f3c540b06d0 con 0x7f3c5406c2b0 2024-03-25T18:16:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.739+0000 7f3c4bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.739+0000 7f3c4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.739+0000 7f3c4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.739+0000 7f3c4bfff640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:16:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.739+0000 7f3c4bfff640 1 -- 172.21.15.204:0/1279543061 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c40003520 con 0x7f3c5406c2b0 2024-03-25T18:16:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.739+0000 7f3c4bfff640 1 -- 172.21.15.204:0/1279543061 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486797015 0 0) 0x7f3c5413feb0 con 0x7f3c54146300 2024-03-25T18:16:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.739+0000 7f3c4bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.739+0000 7f3c4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.739+0000 7f3c4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.739+0000 7f3c4bfff640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:16:32.742 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.739+0000 7f3c4bfff640 1 -- 172.21.15.204:0/1279543061 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c540b06d0 con 0x7f3c54146300 2024-03-25T18:16:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.740+0000 7f3c4bfff640 1 -- 172.21.15.204:0/1279543061 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3979550576 0 0) 0x7f3c5406d050 con 0x7f3c540b0100 2024-03-25T18:16:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.740+0000 7f3c4bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.740+0000 7f3c4bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.740+0000 7f3c4bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.740+0000 7f3c4bfff640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:16:32.743 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.740+0000 7f3c4bfff640 1 -- 172.21.15.204:0/1279543061 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c5413feb0 con 0x7f3c540b0100 2024-03-25T18:16:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.743+0000 7f3c4bfff640 1 -- 172.21.15.204:0/1279543061 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2977243604 0 0) 0x7f3c40003520 con 0x7f3c5406c2b0 2024-03-25T18:16:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.743+0000 7f3c4bfff640 1 -- 172.21.15.204:0/1279543061 >> v1:172.21.15.204:6789/0 conn(0x7f3c540b0100 legacy=0x7f3c54145bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.743+0000 7f3c4bfff640 1 -- 172.21.15.204:0/1279543061 >> v1:172.21.15.122:6789/0 conn(0x7f3c54146300 legacy=0x7f3c5414db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.743+0000 7f3c4bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.743+0000 7f3c4bfff640 1 monclient: found mon.noname-b 2024-03-25T18:16:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.743+0000 7f3c4bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.743+0000 7f3c4bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:16:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.743+0000 7f3c4bfff640 1 -- 172.21.15.204:0/1279543061 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5406d4a0 con 0x7f3c5406c2b0 2024-03-25T18:16:32.744 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.743+0000 7f3c4bfff640 10 monclient: _finish_auth 0 2024-03-25T18:16:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.743+0000 7f3c4bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.743+0000 7f3c4bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.743+0000 7f3c4bfff640 1 -- 172.21.15.204:0/1279543061 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f3c50001690 con 0x7f3c5406c2b0 2024-03-25T18:16:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.743+0000 7f3c4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.743+0000 7f3c4bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.743+0000 7f3c4bfff640 10 monclient: dump: 2024-03-25T18:16:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.745 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.746 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.746 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.743+0000 7f3c5b55d640 5 monclient: authenticate success, global_id 4142 2024-03-25T18:16:32.746 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.743+0000 7f3c5b55d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:32.749 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:32.748+0000 7f4b797b8040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:16:32.770 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.770+0000 7f3c4bfff640 1 -- 172.21.15.204:0/1279543061 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c500017f0 con 0x7f3c5406c2b0 2024-03-25T18:16:32.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.770+0000 7f3c4bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:32.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.770+0000 7f3c4bfff640 1 -- 172.21.15.204:0/1279543061 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f3c50001c00 con 0x7f3c5406c2b0 2024-03-25T18:16:32.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.770+0000 7f3c4bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.770+0000 7f3c4bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:32.771 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.770+0000 7f3c4bfff640 10 monclient: dump: 2024-03-25T18:16:32.771 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.771 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.771 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.771 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.771 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.771 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.771 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.771 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.771 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.772 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.772 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.770+0000 7f3c5b55d640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:32.772 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.770+0000 7f3c5b55d640 4 set_mon_vals no callback set 2024-03-25T18:16:32.775 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.771+0000 7f3c5b55d640 10 monclient: shutdown 2024-03-25T18:16:32.775 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.771+0000 7f3c5b55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:32.775 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.771+0000 7f3c5b55d640 1 -- 172.21.15.204:0/1279543061 >> v1:172.21.15.122:6790/0 conn(0x7f3c5406c2b0 legacy=0x7f3c540af9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.775 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.771+0000 7f3c5b55d640 1 -- 172.21.15.204:0/1279543061 shutdown_connections 2024-03-25T18:16:32.775 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.771+0000 7f3c5b55d640 1 -- 172.21.15.204:0/1279543061 >> 172.21.15.204:0/1279543061 conn(0x7f3c540b5cd0 msgr2=0x7f3c540b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:32.775 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.771+0000 7f3c5b55d640 1 -- 172.21.15.204:0/1279543061 shutdown_connections 2024-03-25T18:16:32.775 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.771+0000 7f3c5b55d640 1 -- 172.21.15.204:0/1279543061 wait complete. 2024-03-25T18:16:32.775 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.772+0000 7f3c5b55d640 10 monclient: build_initial_monmap 2024-03-25T18:16:32.775 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.772+0000 7f3c5b55d640 10 monclient: monmap: 2024-03-25T18:16:32.775 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-25T18:16:32.775 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:32.775 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-03-25T18:16:32.776 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-03-25T18:16:32.776 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:32.776 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-25T18:16:32.776 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:32.776 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:16:32.776 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:16:32.776 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.776 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.772+0000 7f3c5b55d640 1 librados: starting msgr at 2024-03-25T18:16:32.776 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.772+0000 7f3c5b55d640 1 librados: starting objecter 2024-03-25T18:16:32.776 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.772+0000 7f3c5b55d640 1 Processor -- start 2024-03-25T18:16:32.776 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.772+0000 7f3c5b55d640 1 -- start start 2024-03-25T18:16:32.776 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.772+0000 7f3c5b55d640 1 librados: setting wanted keys 2024-03-25T18:16:32.776 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.772+0000 7f3c5b55d640 1 librados: calling monclient init 2024-03-25T18:16:32.776 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.772+0000 7f3c5b55d640 10 monclient: init 2024-03-25T18:16:32.776 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.773+0000 7f3c5b55d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:32.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.773+0000 7f3c5b55d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:32.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.773+0000 7f3c5b55d640 10 monclient(hunting): picked mon.noname-c con 0x7f3c5406c2b0 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:32.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.773+0000 7f3c5b55d640 10 monclient(hunting): picked mon.noname-b con 0x7f3c540b0100 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:32.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.773+0000 7f3c5b55d640 10 monclient(hunting): picked mon.noname-a con 0x7f3c54146300 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:32.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.773+0000 7f3c5b55d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5413c4a0 con 0x7f3c54146300 2024-03-25T18:16:32.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.773+0000 7f3c5b55d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c541427f0 con 0x7f3c540b0100 2024-03-25T18:16:32.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.773+0000 7f3c5b55d640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c54142d10 con 0x7f3c5406c2b0 2024-03-25T18:16:32.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.773+0000 7f3c5b55d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:32.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.773+0000 7f3c5b55d640 10 monclient(hunting): authenticate will time out at 7276.922801s 2024-03-25T18:16:32.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.773+0000 7f3c592f9640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f3c5406c2b0 0x7f3c54137e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.204:51344/0 (socket says 172.21.15.204:51344) 2024-03-25T18:16:32.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.773+0000 7f3c592f9640 1 -- 172.21.15.204:0/1197577569 learned_addr learned my addr 172.21.15.204:0/1197577569 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-25T18:16:32.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.773+0000 7f3c49ffb640 10 client.?.objecter ms_handle_connect 0x7f3c54146300 2024-03-25T18:16:32.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.773+0000 7f3c49ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.777 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.774+0000 7f3c49ffb640 10 client.?.objecter ms_handle_connect 0x7f3c5406c2b0 2024-03-25T18:16:32.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.774+0000 7f3c49ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.774+0000 7f3c49ffb640 10 client.?.objecter ms_handle_connect 0x7f3c540b0100 2024-03-25T18:16:32.778 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.774+0000 7f3c49ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:32.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.775+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874638476 0 0) 0x7f3c5413c4a0 con 0x7f3c54146300 2024-03-25T18:16:32.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.776+0000 7f3c49ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.776+0000 7f3c49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.776+0000 7f3c49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.776+0000 7f3c49ffb640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:16:32.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.776+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c38004eb0 con 0x7f3c54146300 2024-03-25T18:16:32.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.776+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547703680 0 0) 0x7f3c54142d10 con 0x7f3c5406c2b0 2024-03-25T18:16:32.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.776+0000 7f3c49ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.776+0000 7f3c49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.776+0000 7f3c49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.779 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.776+0000 7f3c49ffb640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:16:32.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.776+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c5413c4a0 con 0x7f3c5406c2b0 2024-03-25T18:16:32.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.776+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074052423 0 0) 0x7f3c541427f0 con 0x7f3c540b0100 2024-03-25T18:16:32.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.776+0000 7f3c49ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:32.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.776+0000 7f3c49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:32.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.776+0000 7f3c49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:32.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.776+0000 7f3c49ffb640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:16:32.780 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.776+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c54142d10 con 0x7f3c540b0100 2024-03-25T18:16:32.782 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.780+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 682228516 0 0) 0x7f3c5413c4a0 con 0x7f3c5406c2b0 2024-03-25T18:16:32.782 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.780+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c541427f0 con 0x7f3c5406c2b0 2024-03-25T18:16:32.782 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.780+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2955024833 0 0) 0x7f3c38004eb0 con 0x7f3c54146300 2024-03-25T18:16:32.782 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.780+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c5413c4a0 con 0x7f3c54146300 2024-03-25T18:16:32.782 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.780+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f3c4c001bd0 con 0x7f3c54146300 2024-03-25T18:16:32.782 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.780+0000 7f3c49ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.782 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.780+0000 7f3c49ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:32.782 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.780+0000 7f3c49ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:32.782 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.782 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.783 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.783 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.783 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.783 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.783 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.783 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.783 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.783 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.783 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.780+0000 7f3c49ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:32.783 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.780+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 559852737 0 0) 0x7f3c54142d10 con 0x7f3c540b0100 2024-03-25T18:16:32.783 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.780+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c38004eb0 con 0x7f3c540b0100 2024-03-25T18:16:32.783 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.780+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f3c44001830 con 0x7f3c540b0100 2024-03-25T18:16:32.783 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.780+0000 7f3c49ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.783 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.780+0000 7f3c49ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:32.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.780+0000 7f3c49ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:32.784 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.784 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.784 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.784 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.784 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.784 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.784 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.784 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.784 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.784 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.780+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f3c50001bc0 con 0x7f3c5406c2b0 2024-03-25T18:16:32.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.781+0000 7f3c49ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.781+0000 7f3c49ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:32.784 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.781+0000 7f3c49ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:32.785 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.785 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.785 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.785 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.785 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.785 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.785 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.785 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.785 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.785 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4146826765 0 0) 0x7f3c541427f0 con 0x7f3c5406c2b0 2024-03-25T18:16:32.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 >> v1:172.21.15.204:6789/0 conn(0x7f3c540b0100 legacy=0x7f3c541d7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 >> v1:172.21.15.122:6789/0 conn(0x7f3c54146300 legacy=0x7f3c541e2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:32.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c49ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:32.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c49ffb640 1 monclient: found mon.c 2024-03-25T18:16:32.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c49ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:32.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c49ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:16:32.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c54141cb0 con 0x7f3c5406c2b0 2024-03-25T18:16:32.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c49ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:32.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c49ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:32.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:32.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c5b55d640 5 monclient: authenticate success, global_id 4145 2024-03-25T18:16:32.787 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c5b55d640 10 monclient: _renew_subs 2024-03-25T18:16:32.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c5b55d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:16:32.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c5b55d640 1 -- 172.21.15.204:0/1197577569 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c54138590 con 0x7f3c5406c2b0 2024-03-25T18:16:32.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c5b55d640 10 client.4145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:32.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c5b55d640 10 monclient: _renew_subs 2024-03-25T18:16:32.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c5b55d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:16:32.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c5b55d640 1 -- 172.21.15.204:0/1197577569 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c54144a40 con 0x7f3c5406c2b0 2024-03-25T18:16:32.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.784+0000 7f3c5b55d640 1 librados: init done 2024-03-25T18:16:32.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.785+0000 7f3c5b55d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:32.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.785+0000 7f3c5b55d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:32.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.785+0000 7f3c5b55d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:16:32.788 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.785+0000 7f3c5b55d640 1 -- 172.21.15.204:0/1197577569 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c54138930 con 0x7f3c5406c2b0 2024-03-25T18:16:32.884 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:32.884+0000 7fea77013040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:16:32.971 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.971+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c50004a90 con 0x7f3c5406c2b0 2024-03-25T18:16:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.971+0000 7f3c49ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.971+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f3c50004c00 con 0x7f3c5406c2b0 2024-03-25T18:16:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.971+0000 7f3c49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.971+0000 7f3c49ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.971+0000 7f3c49ffb640 10 monclient: dump: 2024-03-25T18:16:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:32.972 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:32.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.971+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3c500050f0 con 0x7f3c5406c2b0 2024-03-25T18:16:32.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.971+0000 7f3c4b7fe640 4 set_mon_vals no callback set 2024-03-25T18:16:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.973+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 662292788 0 0) 0x7f3c50006730 con 0x7f3c5406c2b0 2024-03-25T18:16:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.973+0000 7f3c49ffb640 10 client.4145.objecter ms_dispatch 0x7f3c54000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:16:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.973+0000 7f3c49ffb640 3 client.4145.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:16:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.973+0000 7f3c49ffb640 3 client.4145.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:16:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.973+0000 7f3c49ffb640 20 client.4145.objecter dump_active .. 0 homeless 2024-03-25T18:16:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.973+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3c500061b0 con 0x7f3c5406c2b0 2024-03-25T18:16:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.973+0000 7f3c49ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:32.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:32.973+0000 7f3c49ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:33.046 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.045+0000 7f3c5b55d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f8aa37f0-78b7-4515-952e-7e25f02c1e3b", "id": 5}] 2024-03-25T18:16:33.046 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.045+0000 7f3c5b55d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f8aa37f0-78b7-4515-952e-7e25f02c1e3b", "id": 5}] 2024-03-25T18:16:33.046 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.045+0000 7f3c5b55d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:16:33.046 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.045+0000 7f3c5b55d640 1 -- 172.21.15.204:0/1197577569 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd new", "uuid": "f8aa37f0-78b7-4515-952e-7e25f02c1e3b", "id": 5} v 0) v1 -- 0x7f3c5404f730 con 0x7f3c5406c2b0 2024-03-25T18:16:33.169 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.169+0000 7f3c49ffb640 1 -- 172.21.15.204:0/1197577569 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f8aa37f0-78b7-4515-952e-7e25f02c1e3b", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3527148154 0 1988412176) 0x7f3c500064a0 con 0x7f3c5406c2b0 2024-03-25T18:16:33.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.169+0000 7f3c49ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f8aa37f0-78b7-4515-952e-7e25f02c1e3b", "id": 5}] 2024-03-25T18:16:33.170 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.169+0000 7f3c49ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:33.170 INFO:teuthology.orchestra.run.smithi204.stdout:5 2024-03-25T18:16:33.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.169+0000 7f3c5b55d640 10 librados: watch_flush enter 2024-03-25T18:16:33.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.170+0000 7f3c5b55d640 10 librados: watch_flush exit 2024-03-25T18:16:33.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.170+0000 7f3c5b55d640 20 client.4145.objecter shutdown clearing up homeless session... 2024-03-25T18:16:33.171 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.170+0000 7f3c5b55d640 10 client.4145.objecter successfully canceled tick 2024-03-25T18:16:33.172 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.170+0000 7f3c5b55d640 10 monclient: shutdown 2024-03-25T18:16:33.172 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.170+0000 7f3c5b55d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:33.172 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.170+0000 7f3c5b55d640 1 -- 172.21.15.204:0/1197577569 >> v1:172.21.15.122:6790/0 conn(0x7f3c5406c2b0 legacy=0x7f3c54137e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:33.172 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.170+0000 7f3c5b55d640 1 -- 172.21.15.204:0/1197577569 shutdown_connections 2024-03-25T18:16:33.172 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.170+0000 7f3c5b55d640 1 -- 172.21.15.204:0/1197577569 >> 172.21.15.204:0/1197577569 conn(0x7f3c540b5cd0 msgr2=0x7f3c540ae6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:33.172 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.170+0000 7f3c5b55d640 1 -- 172.21.15.204:0/1197577569 shutdown_connections 2024-03-25T18:16:33.172 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.170+0000 7f3c5b55d640 1 -- 172.21.15.204:0/1197577569 wait complete. 2024-03-25T18:16:33.172 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.170+0000 7f3c5b55d640 1 librados: shutdown 2024-03-25T18:16:33.180 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new e36ed31b-7bed-4555-979c-e786ae626136 6 2024-03-25T18:16:33.279 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.279+0000 7f9c5ce7b640 10 monclient: get_monmap_and_config 2024-03-25T18:16:33.279 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.279+0000 7f9c5ce7b640 10 monclient: build_initial_monmap 2024-03-25T18:16:33.279 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.279+0000 7f9c5ce7b640 10 monclient: monmap: 2024-03-25T18:16:33.280 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-25T18:16:33.280 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:33.280 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:16:33.279975+0000 2024-03-25T18:16:33.280 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:16:33.279975+0000 2024-03-25T18:16:33.280 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:33.280 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:33.280 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:33.280 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:16:33.280 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:16:33.280 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:33.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.279+0000 7f9c5ce7b640 1 Processor -- start 2024-03-25T18:16:33.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.279+0000 7f9c5ce7b640 1 -- start start 2024-03-25T18:16:33.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.279+0000 7f9c5ce7b640 10 monclient: init 2024-03-25T18:16:33.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.280+0000 7f9c5ce7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:33.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.280+0000 7f9c5ce7b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:33.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.280+0000 7f9c5ce7b640 10 monclient(hunting): picked mon.noname-a con 0x7f9c580b4700 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:33.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.280+0000 7f9c5ce7b640 10 monclient(hunting): picked mon.noname-c con 0x7f9c58142a20 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:33.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.280+0000 7f9c5ce7b640 10 monclient(hunting): picked mon.noname-b con 0x7f9c580a96b0 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:33.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.280+0000 7f9c5ce7b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c580a9d60 con 0x7f9c580b4700 2024-03-25T18:16:33.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.280+0000 7f9c5ce7b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c5813ff10 con 0x7f9c58142a20 2024-03-25T18:16:33.282 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.280+0000 7f9c5ce7b640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c580aa1f0 con 0x7f9c580a96b0 2024-03-25T18:16:33.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.280+0000 7f9c5ce7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:33.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.280+0000 7f9c5ce7b640 10 monclient(hunting): authenticate will time out at 7277.430142s 2024-03-25T18:16:33.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.280+0000 7f9c5659c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9c580b4700 0x7f9c58142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.204:37642/0 (socket says 172.21.15.204:37642) 2024-03-25T18:16:33.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.281+0000 7f9c5659c640 1 -- 172.21.15.204:0/2586744192 learned_addr learned my addr 172.21.15.204:0/2586744192 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-25T18:16:33.283 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.281+0000 7f9c56d9d640 1 --1- 172.21.15.204:0/2586744192 >> v1:172.21.15.122:6790/0 conn(0x7f9c580a96b0 0x7f9c5813f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.204:51348/0 (socket says 172.21.15.204:51348) 2024-03-25T18:16:33.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.283+0000 7f9c55d9b640 1 -- 172.21.15.204:0/2586744192 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498093060 0 0) 0x7f9c5813ff10 con 0x7f9c58142a20 2024-03-25T18:16:33.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.283+0000 7f9c55d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:33.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.283+0000 7f9c55d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:33.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.283+0000 7f9c55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:33.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.283+0000 7f9c55d9b640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:16:33.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.283+0000 7f9c55d9b640 1 -- 172.21.15.204:0/2586744192 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c44003520 con 0x7f9c58142a20 2024-03-25T18:16:33.284 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.283+0000 7f9c55d9b640 1 -- 172.21.15.204:0/2586744192 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233815495 0 0) 0x7f9c580a9d60 con 0x7f9c580b4700 2024-03-25T18:16:33.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.283+0000 7f9c55d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:33.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.283+0000 7f9c55d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:33.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.283+0000 7f9c55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:33.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.283+0000 7f9c55d9b640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:16:33.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.283+0000 7f9c55d9b640 1 -- 172.21.15.204:0/2586744192 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5813ff10 con 0x7f9c580b4700 2024-03-25T18:16:33.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.283+0000 7f9c55d9b640 1 -- 172.21.15.204:0/2586744192 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589032854 0 0) 0x7f9c580aa1f0 con 0x7f9c580a96b0 2024-03-25T18:16:33.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.284+0000 7f9c55d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:33.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.284+0000 7f9c55d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:33.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.284+0000 7f9c55d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:33.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.284+0000 7f9c55d9b640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:16:33.285 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.284+0000 7f9c55d9b640 1 -- 172.21.15.204:0/2586744192 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c580a9d60 con 0x7f9c580a96b0 2024-03-25T18:16:33.288 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.287+0000 7f9c55d9b640 1 -- 172.21.15.204:0/2586744192 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1419038497 0 0) 0x7f9c5813ff10 con 0x7f9c580b4700 2024-03-25T18:16:33.288 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.287+0000 7f9c55d9b640 1 -- 172.21.15.204:0/2586744192 >> v1:172.21.15.122:6790/0 conn(0x7f9c580a96b0 legacy=0x7f9c5813f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:33.288 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.287+0000 7f9c55d9b640 1 -- 172.21.15.204:0/2586744192 >> v1:172.21.15.204:6789/0 conn(0x7f9c58142a20 legacy=0x7f9c5813f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:33.288 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.287+0000 7f9c55d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:16:33.288 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.287+0000 7f9c55d9b640 1 monclient: found mon.noname-a 2024-03-25T18:16:33.288 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.287+0000 7f9c55d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:33.288 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.287+0000 7f9c55d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:16:33.289 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.287+0000 7f9c55d9b640 1 -- 172.21.15.204:0/2586744192 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c580b4ae0 con 0x7f9c580b4700 2024-03-25T18:16:33.289 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.287+0000 7f9c55d9b640 10 monclient: _finish_auth 0 2024-03-25T18:16:33.289 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.287+0000 7f9c55d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:16:33.289 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.287+0000 7f9c55d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:33.289 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.287+0000 7f9c55d9b640 1 -- 172.21.15.204:0/2586744192 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9c48001670 con 0x7f9c580b4700 2024-03-25T18:16:33.289 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.288+0000 7f9c5ce7b640 5 monclient: authenticate success, global_id 4152 2024-03-25T18:16:33.289 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.288+0000 7f9c5ce7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:33.289 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.288+0000 7f9c55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:33.289 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.288+0000 7f9c55d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:33.289 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.288+0000 7f9c55d9b640 10 monclient: dump: 2024-03-25T18:16:33.289 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:33.289 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:33.289 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.289 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.289 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:33.290 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:33.290 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:33.290 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:33.290 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:33.290 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:33.290 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.288+0000 7f9c5ce7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:33.293 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.289+0000 7f9c55d9b640 1 -- 172.21.15.204:0/2586744192 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c480017d0 con 0x7f9c580b4700 2024-03-25T18:16:33.293 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.289+0000 7f9c55d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:33.293 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.289+0000 7f9c55d9b640 1 -- 172.21.15.204:0/2586744192 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9c48001be0 con 0x7f9c580b4700 2024-03-25T18:16:33.293 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.289+0000 7f9c55d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:33.293 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.289+0000 7f9c55d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:33.293 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.289+0000 7f9c55d9b640 10 monclient: dump: 2024-03-25T18:16:33.294 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:33.294 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:33.294 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.294 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.294 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:33.294 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:33.294 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:33.294 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:33.294 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:33.294 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:33.294 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.289+0000 7f9c5ce7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:33.294 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.289+0000 7f9c5ce7b640 4 set_mon_vals no callback set 2024-03-25T18:16:33.294 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.290+0000 7f9c5ce7b640 10 monclient: shutdown 2024-03-25T18:16:33.294 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.290+0000 7f9c5ce7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:33.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.290+0000 7f9c5ce7b640 1 -- 172.21.15.204:0/2586744192 >> v1:172.21.15.122:6789/0 conn(0x7f9c580b4700 legacy=0x7f9c58142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:33.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.290+0000 7f9c5ce7b640 1 -- 172.21.15.204:0/2586744192 shutdown_connections 2024-03-25T18:16:33.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.290+0000 7f9c5ce7b640 1 -- 172.21.15.204:0/2586744192 >> 172.21.15.204:0/2586744192 conn(0x7f9c580668e0 msgr2=0x7f9c58066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:33.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.290+0000 7f9c5ce7b640 1 -- 172.21.15.204:0/2586744192 shutdown_connections 2024-03-25T18:16:33.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.290+0000 7f9c5ce7b640 1 -- 172.21.15.204:0/2586744192 wait complete. 2024-03-25T18:16:33.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.291+0000 7f9c5ce7b640 10 monclient: build_initial_monmap 2024-03-25T18:16:33.295 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.291+0000 7f9c5ce7b640 10 monclient: monmap: 2024-03-25T18:16:33.295 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-25T18:16:33.295 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:33.295 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-03-25T18:16:33.295 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-03-25T18:16:33.295 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:33.295 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-25T18:16:33.295 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:33.296 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:16:33.296 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:16:33.296 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:33.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.291+0000 7f9c5ce7b640 1 librados: starting msgr at 2024-03-25T18:16:33.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.291+0000 7f9c5ce7b640 1 librados: starting objecter 2024-03-25T18:16:33.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.291+0000 7f9c5ce7b640 1 Processor -- start 2024-03-25T18:16:33.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.291+0000 7f9c5ce7b640 1 -- start start 2024-03-25T18:16:33.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.291+0000 7f9c5ce7b640 1 librados: setting wanted keys 2024-03-25T18:16:33.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.291+0000 7f9c5ce7b640 1 librados: calling monclient init 2024-03-25T18:16:33.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.291+0000 7f9c5ce7b640 10 monclient: init 2024-03-25T18:16:33.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.292+0000 7f9c5ce7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:33.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.292+0000 7f9c5ce7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:33.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.292+0000 7f9c5ce7b640 10 monclient(hunting): picked mon.noname-a con 0x7f9c580a96b0 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:33.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.292+0000 7f9c5ce7b640 10 monclient(hunting): picked mon.noname-c con 0x7f9c580b4700 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:33.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.292+0000 7f9c5ce7b640 10 monclient(hunting): picked mon.noname-b con 0x7f9c58142a20 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:33.296 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.292+0000 7f9c5ce7b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c581d7660 con 0x7f9c580a96b0 2024-03-25T18:16:33.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.292+0000 7f9c5ce7b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c581d8200 con 0x7f9c58142a20 2024-03-25T18:16:33.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.292+0000 7f9c5ce7b640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c581d72f0 con 0x7f9c580b4700 2024-03-25T18:16:33.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.292+0000 7f9c5ce7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:33.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.292+0000 7f9c5ce7b640 10 monclient(hunting): authenticate will time out at 7277.441997s 2024-03-25T18:16:33.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.292+0000 7f9c5659c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9c580a96b0 0x7f9c581d7af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.204:37654/0 (socket says 172.21.15.204:37654) 2024-03-25T18:16:33.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.292+0000 7f9c5659c640 1 -- 172.21.15.204:0/163719151 learned_addr learned my addr 172.21.15.204:0/163719151 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-25T18:16:33.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.292+0000 7f9c4effd640 10 client.?.objecter ms_handle_connect 0x7f9c580a96b0 2024-03-25T18:16:33.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.293+0000 7f9c4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:33.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.293+0000 7f9c4effd640 10 client.?.objecter ms_handle_connect 0x7f9c58142a20 2024-03-25T18:16:33.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.293+0000 7f9c4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:33.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.293+0000 7f9c4effd640 10 client.?.objecter ms_handle_connect 0x7f9c580b4700 2024-03-25T18:16:33.297 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.293+0000 7f9c4effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:33.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.294+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 853757664 0 0) 0x7f9c581d7660 con 0x7f9c580a96b0 2024-03-25T18:16:33.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.294+0000 7f9c4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:33.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.294+0000 7f9c4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:33.298 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.294+0000 7f9c4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:33.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.294+0000 7f9c4effd640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:16:33.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.294+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c38004eb0 con 0x7f9c580a96b0 2024-03-25T18:16:33.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.294+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2593756090 0 0) 0x7f9c581d8200 con 0x7f9c58142a20 2024-03-25T18:16:33.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.295+0000 7f9c4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:33.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.295+0000 7f9c4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:33.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.295+0000 7f9c4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:33.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.295+0000 7f9c4effd640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:16:33.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.295+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c581d7660 con 0x7f9c58142a20 2024-03-25T18:16:33.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.295+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1465485869 0 0) 0x7f9c581d72f0 con 0x7f9c580b4700 2024-03-25T18:16:33.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.295+0000 7f9c4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:33.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.295+0000 7f9c4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:33.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.295+0000 7f9c4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:33.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.295+0000 7f9c4effd640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:16:33.299 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.295+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c581d8200 con 0x7f9c580b4700 2024-03-25T18:16:33.300 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.300+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2103937752 0 0) 0x7f9c581d7660 con 0x7f9c58142a20 2024-03-25T18:16:33.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.300+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c581d72f0 con 0x7f9c58142a20 2024-03-25T18:16:33.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.300+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9c50001b50 con 0x7f9c58142a20 2024-03-25T18:16:33.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.300+0000 7f9c4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:33.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.300+0000 7f9c4effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:33.301 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.300+0000 7f9c4effd640 10 monclient(hunting): dump: 2024-03-25T18:16:33.301 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:33.301 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:33.301 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.301 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.301 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:33.301 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:33.301 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:33.301 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:33.301 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:33.302 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:33.302 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.300+0000 7f9c4effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:33.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.301+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 969737311 0 0) 0x7f9c38004eb0 con 0x7f9c580a96b0 2024-03-25T18:16:33.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.301+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c581d7660 con 0x7f9c580a96b0 2024-03-25T18:16:33.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.301+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9c48001c20 con 0x7f9c580a96b0 2024-03-25T18:16:33.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.301+0000 7f9c4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:33.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.301+0000 7f9c4effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:33.303 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.301+0000 7f9c4effd640 10 monclient(hunting): dump: 2024-03-25T18:16:33.303 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:33.303 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:33.303 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.303 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.303 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:33.303 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:33.304 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:33.304 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:33.304 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:33.304 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:33.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.302+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3556279383 0 0) 0x7f9c581d8200 con 0x7f9c580b4700 2024-03-25T18:16:33.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.302+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9c38004eb0 con 0x7f9c580b4700 2024-03-25T18:16:33.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.302+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9c400017b0 con 0x7f9c580b4700 2024-03-25T18:16:33.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.302+0000 7f9c4effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:33.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.302+0000 7f9c4effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:33.304 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.302+0000 7f9c4effd640 10 monclient(hunting): dump: 2024-03-25T18:16:33.304 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:33.304 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:33.305 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.305 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.305 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:33.305 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:33.305 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:33.305 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:33.305 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:33.305 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:33.306 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3990583095 0 0) 0x7f9c581d72f0 con 0x7f9c58142a20 2024-03-25T18:16:33.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 >> v1:172.21.15.122:6790/0 conn(0x7f9c580b4700 legacy=0x7f9c580abaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:33.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 >> v1:172.21.15.122:6789/0 conn(0x7f9c580a96b0 legacy=0x7f9c581d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:33.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c4effd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:33.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c4effd640 1 monclient: found mon.b 2024-03-25T18:16:33.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:33.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c4effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:33.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c580ac200 con 0x7f9c58142a20 2024-03-25T18:16:33.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c4effd640 10 monclient: _finish_auth 0 2024-03-25T18:16:33.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c4effd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:33.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:33.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c5ce7b640 5 monclient: authenticate success, global_id 4153 2024-03-25T18:16:33.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c5ce7b640 10 monclient: _renew_subs 2024-03-25T18:16:33.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c5ce7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:33.307 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c5ce7b640 1 -- 172.21.15.204:0/163719151 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c58136800 con 0x7f9c58142a20 2024-03-25T18:16:33.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c5ce7b640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:33.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c5ce7b640 10 monclient: _renew_subs 2024-03-25T18:16:33.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c5ce7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:33.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c5ce7b640 1 -- 172.21.15.204:0/163719151 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c58133fb0 con 0x7f9c58142a20 2024-03-25T18:16:33.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.304+0000 7f9c5ce7b640 1 librados: init done 2024-03-25T18:16:33.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.305+0000 7f9c5ce7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:33.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.305+0000 7f9c5ce7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:33.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.305+0000 7f9c5ce7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:33.308 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.305+0000 7f9c5ce7b640 1 -- 172.21.15.204:0/163719151 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c581d7480 con 0x7f9c58142a20 2024-03-25T18:16:33.327 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.326+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9c50001fe0 con 0x7f9c58142a20 2024-03-25T18:16:33.327 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.326+0000 7f9c4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:33.327 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.327+0000 7f9c5559a640 4 set_mon_vals no callback set 2024-03-25T18:16:33.334 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.333+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9c50004d00 con 0x7f9c58142a20 2024-03-25T18:16:33.334 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.333+0000 7f9c4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:33.334 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.333+0000 7f9c4effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:33.334 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.333+0000 7f9c4effd640 10 monclient: dump: 2024-03-25T18:16:33.334 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:33.334 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:33.335 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.335 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.335 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:33.335 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:33.335 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:33.335 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:33.335 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:33.335 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:33.335 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.333+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9c500051f0 con 0x7f9c58142a20 2024-03-25T18:16:33.335 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.333+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2049627878 0 0) 0x7f9c50006a10 con 0x7f9c58142a20 2024-03-25T18:16:33.335 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.333+0000 7f9c4effd640 10 client.4153.objecter ms_dispatch 0x7f9c58000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:16:33.335 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.333+0000 7f9c4effd640 3 client.4153.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:16:33.335 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.333+0000 7f9c4effd640 3 client.4153.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:16:33.335 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.334+0000 7f9c4effd640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-25T18:16:33.609 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.609+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9c50006530 con 0x7f9c58142a20 2024-03-25T18:16:33.610 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.609+0000 7f9c4effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:33.610 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.609+0000 7f9c4effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:33.683 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.683+0000 7f9c5ce7b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e36ed31b-7bed-4555-979c-e786ae626136", "id": 6}] 2024-03-25T18:16:33.684 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.683+0000 7f9c5ce7b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e36ed31b-7bed-4555-979c-e786ae626136", "id": 6}] 2024-03-25T18:16:33.684 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.683+0000 7f9c5ce7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:33.684 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.683+0000 7f9c5ce7b640 1 -- 172.21.15.204:0/163719151 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e36ed31b-7bed-4555-979c-e786ae626136", "id": 6} v 0) v1 -- 0x7f9c580adc60 con 0x7f9c58142a20 2024-03-25T18:16:33.709 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:33.709+0000 7f4b797b8040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:16:33.838 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.838+0000 7f9c4effd640 1 -- 172.21.15.204:0/163719151 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e36ed31b-7bed-4555-979c-e786ae626136", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2248811261 0 1113811849) 0x7f9c500054a0 con 0x7f9c58142a20 2024-03-25T18:16:33.838 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.838+0000 7f9c4effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e36ed31b-7bed-4555-979c-e786ae626136", "id": 6}] 2024-03-25T18:16:33.838 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.838+0000 7f9c4effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:33.839 INFO:teuthology.orchestra.run.smithi204.stdout:6 2024-03-25T18:16:33.840 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.839+0000 7f9c5ce7b640 10 librados: watch_flush enter 2024-03-25T18:16:33.840 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.839+0000 7f9c5ce7b640 10 librados: watch_flush exit 2024-03-25T18:16:33.840 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.839+0000 7f9c5ce7b640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-25T18:16:33.840 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.839+0000 7f9c5ce7b640 10 client.4153.objecter successfully canceled tick 2024-03-25T18:16:33.840 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.839+0000 7f9c5ce7b640 10 monclient: shutdown 2024-03-25T18:16:33.840 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.839+0000 7f9c5ce7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:33.841 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.839+0000 7f9c5ce7b640 1 -- 172.21.15.204:0/163719151 >> v1:172.21.15.204:6789/0 conn(0x7f9c58142a20 legacy=0x7f9c581e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:33.841 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.839+0000 7f9c5ce7b640 1 -- 172.21.15.204:0/163719151 shutdown_connections 2024-03-25T18:16:33.841 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.839+0000 7f9c5ce7b640 1 -- 172.21.15.204:0/163719151 >> 172.21.15.204:0/163719151 conn(0x7f9c580668e0 msgr2=0x7f9c580b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:33.841 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.839+0000 7f9c5ce7b640 1 -- 172.21.15.204:0/163719151 shutdown_connections 2024-03-25T18:16:33.841 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.839+0000 7f9c5ce7b640 1 -- 172.21.15.204:0/163719151 wait complete. 2024-03-25T18:16:33.841 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.839+0000 7f9c5ce7b640 1 librados: shutdown 2024-03-25T18:16:33.850 DEBUG:teuthology.orchestra.run.smithi204:> sudo ceph --cluster ceph osd new ad0b583f-133d-4389-80bd-293275a02d0f 7 2024-03-25T18:16:33.874 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:33.875+0000 7fea77013040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:16:33.950 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.949+0000 7f9a7922a640 10 monclient: get_monmap_and_config 2024-03-25T18:16:33.950 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.949+0000 7f9a7922a640 10 monclient: build_initial_monmap 2024-03-25T18:16:33.950 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.949+0000 7f9a7922a640 10 monclient: monmap: 2024-03-25T18:16:33.950 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-25T18:16:33.950 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:33.950 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:16:33.950245+0000 2024-03-25T18:16:33.950 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:16:33.950245+0000 2024-03-25T18:16:33.950 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:33.950 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:33.950 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:33.950 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:16:33.951 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:16:33.951 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:33.952 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.950+0000 7f9a7922a640 1 Processor -- start 2024-03-25T18:16:33.952 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.950+0000 7f9a7922a640 1 -- start start 2024-03-25T18:16:33.952 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.950+0000 7f9a7922a640 10 monclient: init 2024-03-25T18:16:33.953 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.950+0000 7f9a7922a640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:33.953 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.950+0000 7f9a7922a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:33.953 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.950+0000 7f9a7922a640 10 monclient(hunting): picked mon.noname-c con 0x7f9a740b4700 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:33.953 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.950+0000 7f9a7922a640 10 monclient(hunting): picked mon.noname-a con 0x7f9a74142a20 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:33.953 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.950+0000 7f9a7922a640 10 monclient(hunting): picked mon.noname-b con 0x7f9a740a96b0 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:33.953 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.950+0000 7f9a7922a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a740a9d60 con 0x7f9a74142a20 2024-03-25T18:16:33.953 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.950+0000 7f9a7922a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7413ff10 con 0x7f9a740b4700 2024-03-25T18:16:33.953 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.950+0000 7f9a7922a640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a740aa1f0 con 0x7f9a740a96b0 2024-03-25T18:16:33.953 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.950+0000 7f9a7922a640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:33.953 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.950+0000 7f9a7922a640 10 monclient(hunting): authenticate will time out at 7278.100430s 2024-03-25T18:16:33.953 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.951+0000 7f9a6bfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9a74142a20 0x7f9a7413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.204:37660/0 (socket says 172.21.15.204:37660) 2024-03-25T18:16:33.953 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.951+0000 7f9a6bfff640 1 -- 172.21.15.204:0/3811487914 learned_addr learned my addr 172.21.15.204:0/3811487914 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-25T18:16:33.953 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.951+0000 7f9a7359e640 1 --1- 172.21.15.204:0/3811487914 >> v1:172.21.15.122:6790/0 conn(0x7f9a740a96b0 0x7f9a7413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.204:51374/0 (socket says 172.21.15.204:51374) 2024-03-25T18:16:33.954 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.953+0000 7f9a7259c640 1 -- 172.21.15.204:0/3811487914 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331126254 0 0) 0x7f9a7413ff10 con 0x7f9a740b4700 2024-03-25T18:16:33.954 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.953+0000 7f9a7259c640 10 monclient(hunting): _negotiate 2024-03-25T18:16:33.955 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.953+0000 7f9a7259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:33.955 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.953+0000 7f9a7259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:33.955 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.953+0000 7f9a7259c640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:16:33.955 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.953+0000 7f9a7259c640 1 -- 172.21.15.204:0/3811487914 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a60003520 con 0x7f9a740b4700 2024-03-25T18:16:33.955 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.954+0000 7f9a7259c640 1 -- 172.21.15.204:0/3811487914 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3541677106 0 0) 0x7f9a740aa1f0 con 0x7f9a740a96b0 2024-03-25T18:16:33.955 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.954+0000 7f9a7259c640 10 monclient(hunting): _negotiate 2024-03-25T18:16:33.955 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.954+0000 7f9a7259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:33.955 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.954+0000 7f9a7259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:33.955 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.954+0000 7f9a7259c640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:16:33.955 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.954+0000 7f9a7259c640 1 -- 172.21.15.204:0/3811487914 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a7413ff10 con 0x7f9a740a96b0 2024-03-25T18:16:33.955 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.954+0000 7f9a7259c640 1 -- 172.21.15.204:0/3811487914 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582417735 0 0) 0x7f9a740a9d60 con 0x7f9a74142a20 2024-03-25T18:16:33.955 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.954+0000 7f9a7259c640 10 monclient(hunting): _negotiate 2024-03-25T18:16:33.955 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.954+0000 7f9a7259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:33.956 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.954+0000 7f9a7259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:33.956 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.954+0000 7f9a7259c640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:16:33.956 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.954+0000 7f9a7259c640 1 -- 172.21.15.204:0/3811487914 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a740aa1f0 con 0x7f9a74142a20 2024-03-25T18:16:33.958 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7259c640 1 -- 172.21.15.204:0/3811487914 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2412340371 0 0) 0x7f9a60003520 con 0x7f9a740b4700 2024-03-25T18:16:33.958 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7259c640 1 -- 172.21.15.204:0/3811487914 >> v1:172.21.15.122:6790/0 conn(0x7f9a740a96b0 legacy=0x7f9a7413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:33.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7259c640 1 -- 172.21.15.204:0/3811487914 >> v1:172.21.15.122:6789/0 conn(0x7f9a74142a20 legacy=0x7f9a7413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:33.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7259c640 10 monclient: _finish_hunting 0 2024-03-25T18:16:33.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7259c640 1 monclient: found mon.noname-c 2024-03-25T18:16:33.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:33.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7259c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:16:33.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7259c640 1 -- 172.21.15.204:0/3811487914 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a740b4ae0 con 0x7f9a740b4700 2024-03-25T18:16:33.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7259c640 10 monclient: _finish_auth 0 2024-03-25T18:16:33.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7259c640 10 monclient: _check_auth_tickets 2024-03-25T18:16:33.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:33.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7259c640 1 -- 172.21.15.204:0/3811487914 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9a64001690 con 0x7f9a740b4700 2024-03-25T18:16:33.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7922a640 5 monclient: authenticate success, global_id 4156 2024-03-25T18:16:33.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7922a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:33.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:33.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7259c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:33.959 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7259c640 10 monclient: dump: 2024-03-25T18:16:33.960 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:33.960 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:33.960 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.960 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.960 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:33.960 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:33.960 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:33.960 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:33.960 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:33.960 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:33.960 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.957+0000 7f9a7922a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:33.962 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.959+0000 7f9a7259c640 1 -- 172.21.15.204:0/3811487914 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a640017f0 con 0x7f9a740b4700 2024-03-25T18:16:33.962 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.959+0000 7f9a7259c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:33.962 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.959+0000 7f9a7259c640 1 -- 172.21.15.204:0/3811487914 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9a64001c00 con 0x7f9a740b4700 2024-03-25T18:16:33.962 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.959+0000 7f9a7259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:33.962 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.959+0000 7f9a7259c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:33.962 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.959+0000 7f9a7259c640 10 monclient: dump: 2024-03-25T18:16:33.962 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:33.962 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:33.962 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.962 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.962 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:33.963 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:33.963 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:33.963 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:33.963 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:33.963 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:33.963 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.959+0000 7f9a7922a640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:33.963 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.959+0000 7f9a7922a640 4 set_mon_vals no callback set 2024-03-25T18:16:33.963 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.960+0000 7f9a7922a640 10 monclient: shutdown 2024-03-25T18:16:33.963 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.960+0000 7f9a7922a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:33.963 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.960+0000 7f9a7922a640 1 -- 172.21.15.204:0/3811487914 >> v1:172.21.15.204:6789/0 conn(0x7f9a740b4700 legacy=0x7f9a74142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:33.963 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.960+0000 7f9a7922a640 1 -- 172.21.15.204:0/3811487914 shutdown_connections 2024-03-25T18:16:33.963 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.960+0000 7f9a7922a640 1 -- 172.21.15.204:0/3811487914 >> 172.21.15.204:0/3811487914 conn(0x7f9a740668e0 msgr2=0x7f9a74066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:33.963 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.960+0000 7f9a7922a640 1 -- 172.21.15.204:0/3811487914 shutdown_connections 2024-03-25T18:16:33.963 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.960+0000 7f9a7922a640 1 -- 172.21.15.204:0/3811487914 wait complete. 2024-03-25T18:16:33.963 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.961+0000 7f9a7922a640 10 monclient: build_initial_monmap 2024-03-25T18:16:33.964 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.961+0000 7f9a7922a640 10 monclient: monmap: 2024-03-25T18:16:33.964 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 0 2024-03-25T18:16:33.964 INFO:teuthology.orchestra.run.smithi204.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:33.964 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 0.000000 2024-03-25T18:16:33.964 INFO:teuthology.orchestra.run.smithi204.stderr:created 0.000000 2024-03-25T18:16:33.964 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:33.964 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 1 2024-03-25T18:16:33.964 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:33.964 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:16:33.964 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:16:33.964 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:33.964 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.961+0000 7f9a7922a640 1 librados: starting msgr at 2024-03-25T18:16:33.964 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.961+0000 7f9a7922a640 1 librados: starting objecter 2024-03-25T18:16:33.964 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.961+0000 7f9a7922a640 1 Processor -- start 2024-03-25T18:16:33.964 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.961+0000 7f9a7922a640 1 -- start start 2024-03-25T18:16:33.964 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.961+0000 7f9a7922a640 1 librados: setting wanted keys 2024-03-25T18:16:33.965 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.961+0000 7f9a7922a640 1 librados: calling monclient init 2024-03-25T18:16:33.965 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.961+0000 7f9a7922a640 10 monclient: init 2024-03-25T18:16:33.966 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.962+0000 7f9a7922a640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:33.966 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.962+0000 7f9a7922a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:33.966 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.962+0000 7f9a7922a640 10 monclient(hunting): picked mon.noname-c con 0x7f9a740a96b0 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:33.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.962+0000 7f9a7922a640 10 monclient(hunting): picked mon.noname-b con 0x7f9a740b4700 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:33.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.962+0000 7f9a7922a640 10 monclient(hunting): picked mon.noname-a con 0x7f9a74142a20 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:33.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.962+0000 7f9a7922a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a7414eca0 con 0x7f9a74142a20 2024-03-25T18:16:33.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.962+0000 7f9a7922a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a741d74d0 con 0x7f9a740b4700 2024-03-25T18:16:33.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.962+0000 7f9a7922a640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a741d8350 con 0x7f9a740a96b0 2024-03-25T18:16:33.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.962+0000 7f9a7922a640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:33.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.962+0000 7f9a7922a640 10 monclient(hunting): authenticate will time out at 7278.112180s 2024-03-25T18:16:33.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.962+0000 7f9a72d9d640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f9a740a96b0 0x7f9a741d7c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.204:51388/0 (socket says 172.21.15.204:51388) 2024-03-25T18:16:33.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.962+0000 7f9a72d9d640 1 -- 172.21.15.204:0/500752884 learned_addr learned my addr 172.21.15.204:0/500752884 (peer_addr_for_me v1:172.21.15.204:0/0) 2024-03-25T18:16:33.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.963+0000 7f9a6b7fe640 10 client.?.objecter ms_handle_connect 0x7f9a740a96b0 2024-03-25T18:16:33.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.963+0000 7f9a6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:33.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.963+0000 7f9a6b7fe640 10 client.?.objecter ms_handle_connect 0x7f9a74142a20 2024-03-25T18:16:33.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.963+0000 7f9a6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:33.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.963+0000 7f9a6b7fe640 10 client.?.objecter ms_handle_connect 0x7f9a740b4700 2024-03-25T18:16:33.967 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.963+0000 7f9a6b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:33.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719793879 0 0) 0x7f9a741d8350 con 0x7f9a740a96b0 2024-03-25T18:16:33.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:33.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:33.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:33.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:16:33.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a54004eb0 con 0x7f9a740a96b0 2024-03-25T18:16:33.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3744944790 0 0) 0x7f9a7413ff10 con 0x7f9a74142a20 2024-03-25T18:16:33.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:33.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:33.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:33.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:16:33.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a741d8350 con 0x7f9a74142a20 2024-03-25T18:16:33.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2861911538 0 0) 0x7f9a741d74d0 con 0x7f9a740b4700 2024-03-25T18:16:33.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:33.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:33.968 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:33.969 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 10 monclient(hunting): my global_id is 4159 2024-03-25T18:16:33.969 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.965+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a7413ff10 con 0x7f9a740b4700 2024-03-25T18:16:33.970 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.970+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3040918989 0 0) 0x7f9a7413ff10 con 0x7f9a740b4700 2024-03-25T18:16:33.971 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.970+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a741d74d0 con 0x7f9a740b4700 2024-03-25T18:16:33.971 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.970+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9a5c0018f0 con 0x7f9a740b4700 2024-03-25T18:16:33.971 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.970+0000 7f9a6b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:33.971 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.970+0000 7f9a6b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:33.971 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.970+0000 7f9a6b7fe640 10 monclient(hunting): dump: 2024-03-25T18:16:33.971 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:33.971 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:33.971 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.971 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.971 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:33.971 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:33.971 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:33.971 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:33.972 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:33.972 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:33.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.970+0000 7f9a6b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:33.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.971+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2933755376 0 0) 0x7f9a54004eb0 con 0x7f9a740a96b0 2024-03-25T18:16:33.972 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.971+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a7413ff10 con 0x7f9a740a96b0 2024-03-25T18:16:33.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.971+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9a64001bc0 con 0x7f9a740a96b0 2024-03-25T18:16:33.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.971+0000 7f9a6b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:33.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.971+0000 7f9a6b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:33.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.971+0000 7f9a6b7fe640 10 monclient(hunting): dump: 2024-03-25T18:16:33.973 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:33.973 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:33.973 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.973 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.973 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:33.973 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:33.973 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:33.973 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:33.973 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:33.973 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:33.973 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.971+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2590480364 0 0) 0x7f9a741d8350 con 0x7f9a74142a20 2024-03-25T18:16:33.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.971+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a54004eb0 con 0x7f9a74142a20 2024-03-25T18:16:33.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.971+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9a6c0019b0 con 0x7f9a74142a20 2024-03-25T18:16:33.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.971+0000 7f9a6b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:33.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.971+0000 7f9a6b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:33.974 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.971+0000 7f9a6b7fe640 10 monclient(hunting): dump: 2024-03-25T18:16:33.974 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:33.974 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:33.974 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.974 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:33.974 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:33.974 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:33.974 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:33.974 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:33.974 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:33.975 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:33.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2947298727 0 0) 0x7f9a741d74d0 con 0x7f9a740b4700 2024-03-25T18:16:33.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 >> v1:172.21.15.122:6790/0 conn(0x7f9a740a96b0 legacy=0x7f9a741d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:33.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 >> v1:172.21.15.122:6789/0 conn(0x7f9a74142a20 legacy=0x7f9a741d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:33.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a6b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:33.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a6b7fe640 1 monclient: found mon.b 2024-03-25T18:16:33.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a6b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:33.976 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a6b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:33.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a740abd90 con 0x7f9a740b4700 2024-03-25T18:16:33.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a6b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:33.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a6b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:33.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a6b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:33.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a7922a640 5 monclient: authenticate success, global_id 4159 2024-03-25T18:16:33.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a7922a640 10 monclient: _renew_subs 2024-03-25T18:16:33.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a7922a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:33.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a7922a640 1 -- 172.21.15.204:0/500752884 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a741d7720 con 0x7f9a740b4700 2024-03-25T18:16:33.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a7922a640 10 client.4159.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:33.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a7922a640 10 monclient: _renew_subs 2024-03-25T18:16:33.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a7922a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:33.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a7922a640 1 -- 172.21.15.204:0/500752884 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a740ad240 con 0x7f9a740b4700 2024-03-25T18:16:33.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.973+0000 7f9a7922a640 1 librados: init done 2024-03-25T18:16:33.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.974+0000 7f9a7922a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:33.977 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.974+0000 7f9a7922a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:33.978 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.974+0000 7f9a7922a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:33.978 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:33.974+0000 7f9a7922a640 1 -- 172.21.15.204:0/500752884 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a7404f730 con 0x7f9a740b4700 2024-03-25T18:16:34.035 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:34.035+0000 7f4b797b8040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:16:34.077 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.075+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9a5c001d00 con 0x7f9a740b4700 2024-03-25T18:16:34.077 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.076+0000 7f9a6b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:34.077 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.076+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9a5c004980 con 0x7f9a740b4700 2024-03-25T18:16:34.077 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.076+0000 7f9a6b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:34.077 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.076+0000 7f9a6b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:34.077 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.076+0000 7f9a6b7fe640 10 monclient: dump: 2024-03-25T18:16:34.077 INFO:teuthology.orchestra.run.smithi204.stderr:epoch 1 2024-03-25T18:16:34.077 INFO:teuthology.orchestra.run.smithi204.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:34.077 INFO:teuthology.orchestra.run.smithi204.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:34.077 INFO:teuthology.orchestra.run.smithi204.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:34.077 INFO:teuthology.orchestra.run.smithi204.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:34.077 INFO:teuthology.orchestra.run.smithi204.stderr:election_strategy: 3 2024-03-25T18:16:34.077 INFO:teuthology.orchestra.run.smithi204.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:34.078 INFO:teuthology.orchestra.run.smithi204.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:34.078 INFO:teuthology.orchestra.run.smithi204.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:34.078 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:16:34.078 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.076+0000 7f9a71d9b640 4 set_mon_vals no callback set 2024-03-25T18:16:34.078 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.076+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9a5c004e70 con 0x7f9a740b4700 2024-03-25T18:16:34.078 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.076+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 1872881529 0 0) 0x7f9a5c0066a0 con 0x7f9a740b4700 2024-03-25T18:16:34.078 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.076+0000 7f9a6b7fe640 10 client.4159.objecter ms_dispatch 0x7f9a74000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:16:34.078 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.076+0000 7f9a6b7fe640 3 client.4159.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:16:34.078 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.076+0000 7f9a6b7fe640 3 client.4159.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:16:34.078 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.076+0000 7f9a6b7fe640 20 client.4159.objecter dump_active .. 0 homeless 2024-03-25T18:16:34.078 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.076+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9a5c0069c0 con 0x7f9a740b4700 2024-03-25T18:16:34.078 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.076+0000 7f9a6b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:34.078 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.076+0000 7f9a6b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:34.090 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:34.089+0000 7f4b797b8040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:16:34.149 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:34.149+0000 7fea77013040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:16:34.149 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.148+0000 7f9a7922a640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ad0b583f-133d-4389-80bd-293275a02d0f", "id": 7}] 2024-03-25T18:16:34.149 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.148+0000 7f9a7922a640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ad0b583f-133d-4389-80bd-293275a02d0f", "id": 7}] 2024-03-25T18:16:34.149 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.148+0000 7f9a7922a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:34.150 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.148+0000 7f9a7922a640 1 -- 172.21.15.204:0/500752884 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ad0b583f-133d-4389-80bd-293275a02d0f", "id": 7} v 0) v1 -- 0x7f9a740b7470 con 0x7f9a740b4700 2024-03-25T18:16:34.205 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:34.205+0000 7fea77013040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:16:34.208 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.207+0000 7f9a6b7fe640 1 -- 172.21.15.204:0/500752884 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ad0b583f-133d-4389-80bd-293275a02d0f", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 1229913664 0 1371665406) 0x7f9a5c006be0 con 0x7f9a740b4700 2024-03-25T18:16:34.208 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.207+0000 7f9a6b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ad0b583f-133d-4389-80bd-293275a02d0f", "id": 7}] 2024-03-25T18:16:34.208 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.207+0000 7f9a6b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:34.208 INFO:teuthology.orchestra.run.smithi204.stdout:7 2024-03-25T18:16:34.209 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.208+0000 7f9a7922a640 10 librados: watch_flush enter 2024-03-25T18:16:34.209 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.208+0000 7f9a7922a640 10 librados: watch_flush exit 2024-03-25T18:16:34.210 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.208+0000 7f9a7922a640 20 client.4159.objecter shutdown clearing up homeless session... 2024-03-25T18:16:34.210 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.208+0000 7f9a7922a640 10 client.4159.objecter successfully canceled tick 2024-03-25T18:16:34.210 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.208+0000 7f9a7922a640 10 monclient: shutdown 2024-03-25T18:16:34.210 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.208+0000 7f9a7922a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:34.210 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.208+0000 7f9a7922a640 1 -- 172.21.15.204:0/500752884 >> v1:172.21.15.204:6789/0 conn(0x7f9a740b4700 legacy=0x7f9a740acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.210 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.208+0000 7f9a7922a640 1 -- 172.21.15.204:0/500752884 shutdown_connections 2024-03-25T18:16:34.210 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.208+0000 7f9a7922a640 1 -- 172.21.15.204:0/500752884 >> 172.21.15.204:0/500752884 conn(0x7f9a740668e0 msgr2=0x7f9a740b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:34.210 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.208+0000 7f9a7922a640 1 -- 172.21.15.204:0/500752884 shutdown_connections 2024-03-25T18:16:34.210 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.208+0000 7f9a7922a640 1 -- 172.21.15.204:0/500752884 wait complete. 2024-03-25T18:16:34.210 INFO:teuthology.orchestra.run.smithi204.stderr:2024-03-25T18:16:34.209+0000 7f9a7922a640 1 librados: shutdown 2024-03-25T18:16:34.214 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:34.214+0000 7f4b797b8040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:16:34.219 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:16:34.219 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:16:34.220 DEBUG:teuthology.orchestra.run.smithi122:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:16:34.222 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:16:34.222 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:16:34.222 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:16:34.222 DEBUG:teuthology.orchestra.run.smithi122:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:16:34.224 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:16:34.224 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:16:34.225 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:16:34.225 DEBUG:teuthology.orchestra.run.smithi122:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:16:34.227 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:16:34.227 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:16:34.227 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:16:34.227 DEBUG:teuthology.orchestra.run.smithi122:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:16:34.229 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:16:34.229 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:16:34.229 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:16:34.230 DEBUG:teuthology.orchestra.run.smithi204:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:16:34.232 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:16:34.232 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:16:34.232 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:16:34.232 DEBUG:teuthology.orchestra.run.smithi204:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:16:34.234 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:16:34.234 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:16:34.234 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:16:34.235 DEBUG:teuthology.orchestra.run.smithi204:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:16:34.236 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:16:34.237 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:16:34.237 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:16:34.237 DEBUG:teuthology.orchestra.run.smithi204:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:16:34.239 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:16:34.239 DEBUG:teuthology.orchestra.run.smithi122:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:34.394 INFO:tasks.ceph.mgr.y.smithi122.stderr:2024-03-25T18:16:34.394+0000 7fea77013040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:16:34.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.401+0000 7f732478f640 10 monclient: get_monmap_and_config 2024-03-25T18:16:34.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.401+0000 7f732478f640 10 monclient: build_initial_monmap 2024-03-25T18:16:34.401 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.401+0000 7f732478f640 10 monclient: monmap: 2024-03-25T18:16:34.401 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 0 2024-03-25T18:16:34.401 INFO:teuthology.orchestra.run.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:34.401 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-03-25T18:16:34.401581+0000 2024-03-25T18:16:34.402 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-03-25T18:16:34.401581+0000 2024-03-25T18:16:34.402 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:34.402 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:34.402 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:34.402 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:16:34.402 INFO:teuthology.orchestra.run.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:16:34.402 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-25T18:16:34.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.402+0000 7f732478f640 1 Processor -- start 2024-03-25T18:16:34.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.402+0000 7f732478f640 1 -- start start 2024-03-25T18:16:34.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.402+0000 7f732478f640 10 monclient: init 2024-03-25T18:16:34.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.402+0000 7f732478f640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:34.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.402+0000 7f732478f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:34.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.402+0000 7f732478f640 10 monclient(hunting): picked mon.noname-b con 0x7f731c0a9920 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:34.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.402+0000 7f732478f640 10 monclient(hunting): picked mon.noname-a con 0x7f731c0b45a0 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:34.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.402+0000 7f732478f640 10 monclient(hunting): picked mon.noname-c con 0x7f731c146450 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:34.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.402+0000 7f732478f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c06b100 con 0x7f731c0b45a0 2024-03-25T18:16:34.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.403+0000 7f732478f640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c0b2940 con 0x7f731c146450 2024-03-25T18:16:34.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.403+0000 7f732478f640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c13fec0 con 0x7f731c0a9920 2024-03-25T18:16:34.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.403+0000 7f732478f640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:34.403 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.403+0000 7f732478f640 10 monclient(hunting): authenticate will time out at 7317.668002s 2024-03-25T18:16:34.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.403+0000 7f7321d2a640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f731c0b45a0 0x7f731c145d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59326/0 (socket says 172.21.15.122:59326) 2024-03-25T18:16:34.404 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.403+0000 7f7321d2a640 1 -- 172.21.15.122:0/3947587799 learned_addr learned my addr 172.21.15.122:0/3947587799 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:16:34.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.538+0000 7f7321529640 1 -- 172.21.15.122:0/3947587799 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556781785 0 0) 0x7f731c13fec0 con 0x7f731c0a9920 2024-03-25T18:16:34.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.538+0000 7f7321529640 10 monclient(hunting): _negotiate 2024-03-25T18:16:34.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.538+0000 7f7321529640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:34.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.538+0000 7f7321529640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:34.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.538+0000 7f7321529640 10 monclient(hunting): my global_id is 4160 2024-03-25T18:16:34.538 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.538+0000 7f7321529640 1 -- 172.21.15.122:0/3947587799 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7308003520 con 0x7f731c0a9920 2024-03-25T18:16:34.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.543+0000 7f7321529640 1 -- 172.21.15.122:0/3947587799 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1819631472 0 0) 0x7f7308003520 con 0x7f731c0a9920 2024-03-25T18:16:34.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.543+0000 7f7321529640 1 -- 172.21.15.122:0/3947587799 >> v1:172.21.15.204:6789/0 conn(0x7f731c146450 legacy=0x7f731c14dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.543+0000 7f7321529640 1 -- 172.21.15.122:0/3947587799 >> v1:172.21.15.122:6789/0 conn(0x7f731c0b45a0 legacy=0x7f731c145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.543+0000 7f7321529640 10 monclient: _finish_hunting 0 2024-03-25T18:16:34.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.543+0000 7f7321529640 1 monclient: found mon.noname-b 2024-03-25T18:16:34.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.543+0000 7f7321529640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:34.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.543+0000 7f7321529640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:16:34.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.543+0000 7f7321529640 1 -- 172.21.15.122:0/3947587799 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c140050 con 0x7f731c0a9920 2024-03-25T18:16:34.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.543+0000 7f7321529640 10 monclient: _finish_auth 0 2024-03-25T18:16:34.543 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.543+0000 7f7321529640 10 monclient: _check_auth_tickets 2024-03-25T18:16:34.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.543+0000 7f7321529640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:34.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.543+0000 7f732478f640 5 monclient: authenticate success, global_id 4160 2024-03-25T18:16:34.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.543+0000 7f732478f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:34.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.544+0000 7f7321529640 1 -- 172.21.15.122:0/3947587799 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f73180014f0 con 0x7f731c0a9920 2024-03-25T18:16:34.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.544+0000 7f7321529640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:34.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.544+0000 7f7321529640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:34.544 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.544+0000 7f7321529640 10 monclient: dump: 2024-03-25T18:16:34.544 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-03-25T18:16:34.544 INFO:teuthology.orchestra.run.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:34.544 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:34.545 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:34.545 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:34.545 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:34.545 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:34.545 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:34.545 INFO:teuthology.orchestra.run.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:34.545 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-25T18:16:34.545 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.544+0000 7f732478f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:34.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.545+0000 7f7321529640 1 -- 172.21.15.122:0/3947587799 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7318001780 con 0x7f731c0a9920 2024-03-25T18:16:34.546 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.546+0000 7f7321529640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:34.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.546+0000 7f7321529640 1 -- 172.21.15.122:0/3947587799 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f7318001a90 con 0x7f731c0a9920 2024-03-25T18:16:34.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.546+0000 7f7321529640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:34.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.546+0000 7f7321529640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:34.547 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.546+0000 7f7321529640 10 monclient: dump: 2024-03-25T18:16:34.547 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-03-25T18:16:34.547 INFO:teuthology.orchestra.run.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:34.547 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:34.547 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:34.547 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:34.547 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:34.547 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:34.547 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:34.547 INFO:teuthology.orchestra.run.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:34.547 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-25T18:16:34.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.546+0000 7f732478f640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:34.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.546+0000 7f732478f640 4 set_mon_vals no callback set 2024-03-25T18:16:34.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.546+0000 7f732478f640 10 monclient: shutdown 2024-03-25T18:16:34.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.546+0000 7f732478f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:34.548 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.546+0000 7f732478f640 1 -- 172.21.15.122:0/3947587799 >> v1:172.21.15.122:6790/0 conn(0x7f731c0a9920 legacy=0x7f731c13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.550 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.547+0000 7f732478f640 1 -- 172.21.15.122:0/3947587799 shutdown_connections 2024-03-25T18:16:34.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.547+0000 7f732478f640 1 -- 172.21.15.122:0/3947587799 >> 172.21.15.122:0/3947587799 conn(0x7f731c0b7f40 msgr2=0x7f731c0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:34.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.547+0000 7f732478f640 1 -- 172.21.15.122:0/3947587799 shutdown_connections 2024-03-25T18:16:34.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.547+0000 7f732478f640 1 -- 172.21.15.122:0/3947587799 wait complete. 2024-03-25T18:16:34.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.548+0000 7f732478f640 10 monclient: build_initial_monmap 2024-03-25T18:16:34.551 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.548+0000 7f732478f640 10 monclient: monmap: 2024-03-25T18:16:34.551 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 0 2024-03-25T18:16:34.551 INFO:teuthology.orchestra.run.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:34.551 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 0.000000 2024-03-25T18:16:34.551 INFO:teuthology.orchestra.run.smithi122.stderr:created 0.000000 2024-03-25T18:16:34.551 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:34.551 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 1 2024-03-25T18:16:34.551 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:34.551 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:16:34.551 INFO:teuthology.orchestra.run.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:16:34.551 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-25T18:16:34.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.548+0000 7f732478f640 1 librados: starting msgr at 2024-03-25T18:16:34.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.548+0000 7f732478f640 1 librados: starting objecter 2024-03-25T18:16:34.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.548+0000 7f732478f640 1 Processor -- start 2024-03-25T18:16:34.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.548+0000 7f732478f640 1 -- start start 2024-03-25T18:16:34.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.548+0000 7f732478f640 1 librados: setting wanted keys 2024-03-25T18:16:34.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.548+0000 7f732478f640 1 librados: calling monclient init 2024-03-25T18:16:34.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.548+0000 7f732478f640 10 monclient: init 2024-03-25T18:16:34.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.549+0000 7f732478f640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:34.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.549+0000 7f732478f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:34.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.549+0000 7f732478f640 10 monclient(hunting): picked mon.noname-a con 0x7f731c0a9920 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:34.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.549+0000 7f732478f640 10 monclient(hunting): picked mon.noname-b con 0x7f731c0b45a0 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:34.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.549+0000 7f732478f640 10 monclient(hunting): picked mon.noname-c con 0x7f731c146450 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:34.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.549+0000 7f732478f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c1e3e30 con 0x7f731c0a9920 2024-03-25T18:16:34.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.549+0000 7f732478f640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c1d85c0 con 0x7f731c0b45a0 2024-03-25T18:16:34.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.549+0000 7f732478f640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731c141550 con 0x7f731c146450 2024-03-25T18:16:34.552 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.549+0000 7f732478f640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:34.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.549+0000 7f732478f640 10 monclient(hunting): authenticate will time out at 7317.814697s 2024-03-25T18:16:34.553 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.552+0000 7f7322d2c640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f731c146450 0x7f731c1d7eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:48518/0 (socket says 172.21.15.122:48518) 2024-03-25T18:16:34.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.552+0000 7f7322d2c640 1 -- 172.21.15.122:0/3151088262 learned_addr learned my addr 172.21.15.122:0/3151088262 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:16:34.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.553+0000 7f7312ffd640 10 client.?.objecter ms_handle_connect 0x7f731c146450 2024-03-25T18:16:34.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.553+0000 7f7312ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:34.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.553+0000 7f7312ffd640 10 client.?.objecter ms_handle_connect 0x7f731c0b45a0 2024-03-25T18:16:34.554 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.553+0000 7f7312ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:34.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.575+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4027665835 0 0) 0x7f731c141550 con 0x7f731c146450 2024-03-25T18:16:34.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.575+0000 7f7312ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:34.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.575+0000 7f7312ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:34.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.575+0000 7f7312ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:34.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.575+0000 7f7312ffd640 10 monclient(hunting): my global_id is 4163 2024-03-25T18:16:34.575 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.575+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7300004eb0 con 0x7f731c146450 2024-03-25T18:16:34.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.579+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22130748 0 0) 0x7f731c1d85c0 con 0x7f731c0b45a0 2024-03-25T18:16:34.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.579+0000 7f7312ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:34.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.579+0000 7f7312ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:34.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.579+0000 7f7312ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:34.579 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.579+0000 7f7312ffd640 10 monclient(hunting): my global_id is 4162 2024-03-25T18:16:34.580 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.579+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f731c141550 con 0x7f731c0b45a0 2024-03-25T18:16:34.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.585+0000 7f7312ffd640 10 client.?.objecter ms_handle_connect 0x7f731c0a9920 2024-03-25T18:16:34.585 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.585+0000 7f7312ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:34.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.589+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 551917868 0 0) 0x7f731c141550 con 0x7f731c0b45a0 2024-03-25T18:16:34.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.590+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f731c1d85c0 con 0x7f731c0b45a0 2024-03-25T18:16:34.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.590+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f730c0018a0 con 0x7f731c0b45a0 2024-03-25T18:16:34.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.590+0000 7f7312ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:34.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.590+0000 7f7312ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:34.590 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.590+0000 7f7312ffd640 10 monclient(hunting): dump: 2024-03-25T18:16:34.590 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-03-25T18:16:34.590 INFO:teuthology.orchestra.run.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:34.590 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:34.591 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:34.591 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:34.591 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:34.591 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:34.591 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:34.591 INFO:teuthology.orchestra.run.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:34.591 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-25T18:16:34.591 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.590+0000 7f7312ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:34.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.594+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2324351305 0 0) 0x7f731c1e3e30 con 0x7f731c0a9920 2024-03-25T18:16:34.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.594+0000 7f7312ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:34.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.594+0000 7f7312ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:34.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.594+0000 7f7312ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:34.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.594+0000 7f7312ffd640 10 monclient(hunting): my global_id is 4167 2024-03-25T18:16:34.594 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.594+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f731c141550 con 0x7f731c0a9920 2024-03-25T18:16:34.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2729828850 0 0) 0x7f731c1d85c0 con 0x7f731c0b45a0 2024-03-25T18:16:34.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 >> v1:172.21.15.122:6790/0 conn(0x7f731c146450 legacy=0x7f731c1d7eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 >> v1:172.21.15.122:6789/0 conn(0x7f731c0a9920 legacy=0x7f731c1e42c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:34.596 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f7312ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:34.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f7312ffd640 1 monclient: found mon.b 2024-03-25T18:16:34.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f7312ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:34.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f7312ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:34.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c13bd80 con 0x7f731c0b45a0 2024-03-25T18:16:34.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f7312ffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:34.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f7312ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:34.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f7312ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:34.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f732478f640 5 monclient: authenticate success, global_id 4162 2024-03-25T18:16:34.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f732478f640 10 monclient: _renew_subs 2024-03-25T18:16:34.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f732478f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:34.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f732478f640 1 -- 172.21.15.122:0/3151088262 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f731c141c50 con 0x7f731c0b45a0 2024-03-25T18:16:34.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f732478f640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:34.597 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f732478f640 10 monclient: _renew_subs 2024-03-25T18:16:34.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f732478f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:34.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f732478f640 1 -- 172.21.15.122:0/3151088262 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f731c1e4bb0 con 0x7f731c0b45a0 2024-03-25T18:16:34.598 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.595+0000 7f732478f640 1 librados: init done 2024-03-25T18:16:34.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.597+0000 7f732478f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:34.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.597+0000 7f732478f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:34.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.597+0000 7f732478f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:34.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.597+0000 7f732478f640 1 -- 172.21.15.122:0/3151088262 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f731c04f730 con 0x7f731c0b45a0 2024-03-25T18:16:34.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.598+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f730c0047e0 con 0x7f731c0b45a0 2024-03-25T18:16:34.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.598+0000 7f7312ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:34.599 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.598+0000 7f7320d28640 4 set_mon_vals no callback set 2024-03-25T18:16:34.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.639+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f730c004bb0 con 0x7f731c0b45a0 2024-03-25T18:16:34.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.639+0000 7f7312ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:34.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.639+0000 7f7312ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:34.639 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.639+0000 7f7312ffd640 10 monclient: dump: 2024-03-25T18:16:34.639 INFO:teuthology.orchestra.run.smithi122.stderr:epoch 1 2024-03-25T18:16:34.640 INFO:teuthology.orchestra.run.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:34.640 INFO:teuthology.orchestra.run.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:34.640 INFO:teuthology.orchestra.run.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:34.640 INFO:teuthology.orchestra.run.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:34.640 INFO:teuthology.orchestra.run.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:34.640 INFO:teuthology.orchestra.run.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:34.640 INFO:teuthology.orchestra.run.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:34.640 INFO:teuthology.orchestra.run.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:34.640 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-25T18:16:34.793 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.793+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 1024756320 0 0) 0x7f730c004d90 con 0x7f731c0b45a0 2024-03-25T18:16:34.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.798+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f730c015690 con 0x7f731c0b45a0 2024-03-25T18:16:34.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.798+0000 7f7312ffd640 10 client.4162.objecter ms_dispatch 0x7f731c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:34.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.798+0000 7f7312ffd640 3 client.4162.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:34.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.798+0000 7f7312ffd640 3 client.4162.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:34.798 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.798+0000 7f7312ffd640 20 client.4162.objecter dump_active .. 0 homeless 2024-03-25T18:16:34.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.798+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f730c014950 con 0x7f731c0b45a0 2024-03-25T18:16:34.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.798+0000 7f7312ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:34.799 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.798+0000 7f7312ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:34.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.918+0000 7f732478f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:34.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.918+0000 7f732478f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:34.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.918+0000 7f732478f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:34.918 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:34.918+0000 7f732478f640 1 -- 172.21.15.122:0/3151088262 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f731c069f30 con 0x7f731c0b45a0 2024-03-25T18:16:35.040 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:35.040+0000 7f7312ffd640 1 -- 172.21.15.122:0/3151088262 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f730c014f90 con 0x7f731c0b45a0 2024-03-25T18:16:35.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:35.040+0000 7f7312ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:35.041 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:35.040+0000 7f7312ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:35.041 INFO:teuthology.orchestra.run.smithi122.stdout: 2024-03-25T18:16:35.041 INFO:teuthology.orchestra.run.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:35.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:35.041+0000 7f732478f640 10 librados: watch_flush enter 2024-03-25T18:16:35.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:35.041+0000 7f732478f640 10 librados: watch_flush exit 2024-03-25T18:16:35.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:35.041+0000 7f732478f640 20 client.4162.objecter shutdown clearing up homeless session... 2024-03-25T18:16:35.042 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:35.041+0000 7f732478f640 10 client.4162.objecter successfully canceled tick 2024-03-25T18:16:35.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:35.041+0000 7f732478f640 10 monclient: shutdown 2024-03-25T18:16:35.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:35.041+0000 7f732478f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:35.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:35.041+0000 7f732478f640 1 -- 172.21.15.122:0/3151088262 >> v1:172.21.15.204:6789/0 conn(0x7f731c0b45a0 legacy=0x7f731c1d77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:35.042+0000 7f732478f640 1 -- 172.21.15.122:0/3151088262 shutdown_connections 2024-03-25T18:16:35.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:35.042+0000 7f732478f640 1 -- 172.21.15.122:0/3151088262 >> 172.21.15.122:0/3151088262 conn(0x7f731c0b7f40 msgr2=0x7f731c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:35.043 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:35.042+0000 7f732478f640 1 -- 172.21.15.122:0/3151088262 shutdown_connections 2024-03-25T18:16:35.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:35.042+0000 7f732478f640 1 -- 172.21.15.122:0/3151088262 wait complete. 2024-03-25T18:16:35.044 INFO:teuthology.orchestra.run.smithi122.stderr:2024-03-25T18:16:35.042+0000 7f732478f640 1 librados: shutdown 2024-03-25T18:16:35.057 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:16:35.057 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:16:35.862 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:36.006 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.005+0000 7fcf99c4a640 10 monclient: get_monmap_and_config 2024-03-25T18:16:36.006 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.005+0000 7fcf99c4a640 10 monclient: build_initial_monmap 2024-03-25T18:16:36.006 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.005+0000 7fcf99c4a640 10 monclient: monmap: 2024-03-25T18:16:36.006 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:16:36.006 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:36.006 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:16:36.006211+0000 2024-03-25T18:16:36.006 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:16:36.006211+0000 2024-03-25T18:16:36.006 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:36.006 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:36.006 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:36.006 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:16:36.006 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:16:36.006 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:36.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.006+0000 7fcf99c4a640 1 Processor -- start 2024-03-25T18:16:36.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.006+0000 7fcf99c4a640 1 -- start start 2024-03-25T18:16:36.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.006+0000 7fcf99c4a640 10 monclient: init 2024-03-25T18:16:36.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.007+0000 7fcf99c4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:36.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.007+0000 7fcf99c4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:36.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.007+0000 7fcf99c4a640 10 monclient(hunting): picked mon.noname-c con 0x7fcf940a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:36.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.007+0000 7fcf99c4a640 10 monclient(hunting): picked mon.noname-a con 0x7fcf9413f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:36.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.008+0000 7fcf99c4a640 10 monclient(hunting): picked mon.noname-b con 0x7fcf941430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:36.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.008+0000 7fcf99c4a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf94142520 con 0x7fcf9413f710 2024-03-25T18:16:36.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.008+0000 7fcf99c4a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf940b28a0 con 0x7fcf940a98f0 2024-03-25T18:16:36.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.008+0000 7fcf99c4a640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf940b2a30 con 0x7fcf941430c0 2024-03-25T18:16:36.010 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.008+0000 7fcf99c4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:36.010 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.008+0000 7fcf99c4a640 10 monclient(hunting): authenticate will time out at 7319.274502s 2024-03-25T18:16:36.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.016+0000 7fcf93fff640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7fcf941430c0 0x7fcf9414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:48532/0 (socket says 172.21.15.122:48532) 2024-03-25T18:16:36.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.016+0000 7fcf93fff640 1 -- 172.21.15.122:0/441852258 learned_addr learned my addr 172.21.15.122:0/441852258 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:16:36.022 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:36.022+0000 7f4b1ac17640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:16:36.075 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:16:36.074+0000 7f4b1ac17640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:16:36.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.193+0000 7fcf927fc640 1 -- 172.21.15.122:0/441852258 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888441561 0 0) 0x7fcf940b28a0 con 0x7fcf940a98f0 2024-03-25T18:16:36.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.193+0000 7fcf927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:36.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.193+0000 7fcf927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:36.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.193+0000 7fcf927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:36.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.193+0000 7fcf927fc640 10 monclient(hunting): my global_id is 4180 2024-03-25T18:16:36.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.193+0000 7fcf927fc640 1 -- 172.21.15.122:0/441852258 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf7c003520 con 0x7fcf940a98f0 2024-03-25T18:16:36.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.198+0000 7fcf927fc640 1 -- 172.21.15.122:0/441852258 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3223173659 0 0) 0x7fcf7c003520 con 0x7fcf940a98f0 2024-03-25T18:16:36.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.198+0000 7fcf927fc640 1 -- 172.21.15.122:0/441852258 >> v1:172.21.15.122:6790/0 conn(0x7fcf941430c0 legacy=0x7fcf9414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:36.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.198+0000 7fcf927fc640 1 -- 172.21.15.122:0/441852258 >> v1:172.21.15.122:6789/0 conn(0x7fcf9413f710 legacy=0x7fcf941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:36.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.198+0000 7fcf927fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:36.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.198+0000 7fcf927fc640 1 monclient: found mon.noname-c 2024-03-25T18:16:36.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.198+0000 7fcf927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:36.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.198+0000 7fcf927fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:16:36.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.198+0000 7fcf927fc640 1 -- 172.21.15.122:0/441852258 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf940b44c0 con 0x7fcf940a98f0 2024-03-25T18:16:36.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.198+0000 7fcf927fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:36.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.198+0000 7fcf927fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:36.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.198+0000 7fcf927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:36.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.198+0000 7fcf927fc640 1 -- 172.21.15.122:0/441852258 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fcf88001690 con 0x7fcf940a98f0 2024-03-25T18:16:36.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.198+0000 7fcf99c4a640 5 monclient: authenticate success, global_id 4180 2024-03-25T18:16:36.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.198+0000 7fcf927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:36.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.198+0000 7fcf927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:36.201 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.198+0000 7fcf927fc640 10 monclient: dump: 2024-03-25T18:16:36.201 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:36.201 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:36.201 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:36.201 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:36.201 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:36.201 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:36.201 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:36.201 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:36.201 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:36.201 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:36.201 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.198+0000 7fcf99c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:36.209 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.207+0000 7fcf927fc640 1 -- 172.21.15.122:0/441852258 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf880017f0 con 0x7fcf940a98f0 2024-03-25T18:16:36.209 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.207+0000 7fcf927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:36.209 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.207+0000 7fcf927fc640 1 -- 172.21.15.122:0/441852258 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fcf88001c00 con 0x7fcf940a98f0 2024-03-25T18:16:36.209 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.207+0000 7fcf927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:36.209 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.207+0000 7fcf927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:36.209 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.207+0000 7fcf927fc640 10 monclient: dump: 2024-03-25T18:16:36.209 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:36.209 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:36.209 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:36.209 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:36.209 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:36.209 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:36.209 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:36.209 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:36.209 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:36.209 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:36.210 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.207+0000 7fcf99c4a640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:36.210 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.207+0000 7fcf99c4a640 4 set_mon_vals no callback set 2024-03-25T18:16:36.210 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.208+0000 7fcf99c4a640 10 monclient: shutdown 2024-03-25T18:16:36.210 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.208+0000 7fcf99c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:36.210 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.208+0000 7fcf99c4a640 1 -- 172.21.15.122:0/441852258 >> v1:172.21.15.204:6789/0 conn(0x7fcf940a98f0 legacy=0x7fcf940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:36.211 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.208+0000 7fcf99c4a640 1 -- 172.21.15.122:0/441852258 shutdown_connections 2024-03-25T18:16:36.211 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.208+0000 7fcf99c4a640 1 -- 172.21.15.122:0/441852258 >> 172.21.15.122:0/441852258 conn(0x7fcf940b7ea0 msgr2=0x7fcf940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:36.211 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.209+0000 7fcf99c4a640 1 -- 172.21.15.122:0/441852258 shutdown_connections 2024-03-25T18:16:36.211 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.209+0000 7fcf99c4a640 1 -- 172.21.15.122:0/441852258 wait complete. 2024-03-25T18:16:36.211 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.210+0000 7fcf99c4a640 10 monclient: build_initial_monmap 2024-03-25T18:16:36.211 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.210+0000 7fcf99c4a640 10 monclient: monmap: 2024-03-25T18:16:36.211 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:16:36.211 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:36.211 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:16:36.211 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:16:36.211 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:36.211 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:16:36.212 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:36.212 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:16:36.212 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:16:36.212 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:36.213 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.210+0000 7fcf99c4a640 1 librados: starting msgr at 2024-03-25T18:16:36.213 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.210+0000 7fcf99c4a640 1 librados: starting objecter 2024-03-25T18:16:36.213 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.211+0000 7fcf99c4a640 1 Processor -- start 2024-03-25T18:16:36.213 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.211+0000 7fcf99c4a640 1 -- start start 2024-03-25T18:16:36.213 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.211+0000 7fcf99c4a640 1 librados: setting wanted keys 2024-03-25T18:16:36.213 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.211+0000 7fcf99c4a640 1 librados: calling monclient init 2024-03-25T18:16:36.213 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.211+0000 7fcf99c4a640 10 monclient: init 2024-03-25T18:16:36.214 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.212+0000 7fcf99c4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:36.214 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.212+0000 7fcf99c4a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:36.214 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.212+0000 7fcf99c4a640 10 monclient(hunting): picked mon.noname-c con 0x7fcf940a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:36.214 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.212+0000 7fcf99c4a640 10 monclient(hunting): picked mon.noname-b con 0x7fcf9413f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:36.214 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.213+0000 7fcf99c4a640 10 monclient(hunting): picked mon.noname-a con 0x7fcf941430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:36.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.213+0000 7fcf99c4a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf941db7d0 con 0x7fcf941430c0 2024-03-25T18:16:36.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.213+0000 7fcf99c4a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf940b1420 con 0x7fcf9413f710 2024-03-25T18:16:36.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.213+0000 7fcf99c4a640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf940b15b0 con 0x7fcf940a98f0 2024-03-25T18:16:36.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.213+0000 7fcf99c4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:36.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.213+0000 7fcf99c4a640 10 monclient(hunting): authenticate will time out at 7319.479509s 2024-03-25T18:16:36.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.213+0000 7fcf93fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fcf941430c0 0x7fcf941d4350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59344/0 (socket says 172.21.15.122:59344) 2024-03-25T18:16:36.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.213+0000 7fcf93fff640 1 -- 172.21.15.122:0/552939038 learned_addr learned my addr 172.21.15.122:0/552939038 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:16:36.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.214+0000 7fcf7bfff640 10 client.?.objecter ms_handle_connect 0x7fcf941430c0 2024-03-25T18:16:36.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.214+0000 7fcf7bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:36.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.214+0000 7fcf7bfff640 10 client.?.objecter ms_handle_connect 0x7fcf9413f710 2024-03-25T18:16:36.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.214+0000 7fcf7bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:36.218 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.217+0000 7fcf7bfff640 1 -- 172.21.15.122:0/552939038 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136334624 0 0) 0x7fcf941db7d0 con 0x7fcf941430c0 2024-03-25T18:16:36.218 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.217+0000 7fcf7bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:36.219 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.217+0000 7fcf7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:36.219 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.217+0000 7fcf7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:36.219 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.217+0000 7fcf7bfff640 10 monclient(hunting): my global_id is 4182 2024-03-25T18:16:36.219 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.217+0000 7fcf7bfff640 1 -- 172.21.15.122:0/552939038 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf80004eb0 con 0x7fcf941430c0 2024-03-25T18:16:36.228 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.226+0000 7fcf7bfff640 1 -- 172.21.15.122:0/552939038 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4190990543 0 0) 0x7fcf80004eb0 con 0x7fcf941430c0 2024-03-25T18:16:36.228 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.227+0000 7fcf7bfff640 1 -- 172.21.15.122:0/552939038 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf941db7d0 con 0x7fcf941430c0 2024-03-25T18:16:36.228 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.227+0000 7fcf7bfff640 1 -- 172.21.15.122:0/552939038 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fcf8c001b10 con 0x7fcf941430c0 2024-03-25T18:16:36.228 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.227+0000 7fcf7bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:36.228 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.227+0000 7fcf7bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:36.228 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.227+0000 7fcf7bfff640 10 monclient(hunting): dump: 2024-03-25T18:16:36.228 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:36.228 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:36.228 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:36.228 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:36.228 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:36.228 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:36.228 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:36.228 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:36.229 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:36.229 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:36.229 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.227+0000 7fcf7bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:36.233 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf7bfff640 1 -- 172.21.15.122:0/552939038 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2011576830 0 0) 0x7fcf941db7d0 con 0x7fcf941430c0 2024-03-25T18:16:36.233 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf7bfff640 1 -- 172.21.15.122:0/552939038 >> v1:172.21.15.122:6790/0 conn(0x7fcf940a98f0 legacy=0x7fcf941daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:36.233 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf7bfff640 1 -- 172.21.15.122:0/552939038 >> v1:172.21.15.204:6789/0 conn(0x7fcf9413f710 legacy=0x7fcf940b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:36.233 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf7bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:36.233 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf7bfff640 1 monclient: found mon.a 2024-03-25T18:16:36.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:36.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf7bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:36.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf7bfff640 1 -- 172.21.15.122:0/552939038 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf940b1740 con 0x7fcf941430c0 2024-03-25T18:16:36.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf7bfff640 10 monclient: _finish_auth 0 2024-03-25T18:16:36.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf7bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:36.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:36.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf99c4a640 5 monclient: authenticate success, global_id 4182 2024-03-25T18:16:36.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf99c4a640 10 monclient: _renew_subs 2024-03-25T18:16:36.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf99c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:36.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf99c4a640 1 -- 172.21.15.122:0/552939038 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf941db1a0 con 0x7fcf941430c0 2024-03-25T18:16:36.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf99c4a640 10 client.4182.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:36.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf99c4a640 10 monclient: _renew_subs 2024-03-25T18:16:36.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf99c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:36.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf99c4a640 1 -- 172.21.15.122:0/552939038 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf940b08f0 con 0x7fcf941430c0 2024-03-25T18:16:36.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.232+0000 7fcf99c4a640 1 librados: init done 2024-03-25T18:16:36.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.234+0000 7fcf99c4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:36.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.234+0000 7fcf99c4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:36.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.234+0000 7fcf99c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:36.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.234+0000 7fcf99c4a640 1 -- 172.21.15.122:0/552939038 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf940627b0 con 0x7fcf941430c0 2024-03-25T18:16:36.260 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.259+0000 7fcf7bfff640 1 -- 172.21.15.122:0/552939038 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf8c002080 con 0x7fcf941430c0 2024-03-25T18:16:36.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.259+0000 7fcf7bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:36.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.259+0000 7fcf7bfff640 1 -- 172.21.15.122:0/552939038 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fcf8c004d00 con 0x7fcf941430c0 2024-03-25T18:16:36.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.259+0000 7fcf7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:36.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.259+0000 7fcf7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:36.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.259+0000 7fcf7bfff640 10 monclient: dump: 2024-03-25T18:16:36.261 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:36.261 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:36.261 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:36.261 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:36.261 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:36.261 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:36.261 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:36.261 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:36.261 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:36.261 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:36.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.259+0000 7fcf91ffb640 4 set_mon_vals no callback set 2024-03-25T18:16:36.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.262+0000 7fcf7bfff640 1 -- 172.21.15.122:0/552939038 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 2540185285 0 0) 0x7fcf8c004f20 con 0x7fcf941430c0 2024-03-25T18:16:36.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.263+0000 7fcf937fe640 10 monclient: get_auth_request con 0x7fcf8006dac0 auth_method 0 2024-03-25T18:16:36.264 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.263+0000 7fcf7bfff640 10 client.4182.objecter ms_handle_connect 0x7fcf8006dac0 2024-03-25T18:16:36.268 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.267+0000 7fcf7bfff640 1 -- 172.21.15.122:0/552939038 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fcf8c054a90 con 0x7fcf941430c0 2024-03-25T18:16:36.269 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.267+0000 7fcf7bfff640 10 client.4182.objecter ms_dispatch 0x7fcf94000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:36.269 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.267+0000 7fcf7bfff640 3 client.4182.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:36.269 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.267+0000 7fcf7bfff640 3 client.4182.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:36.269 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.267+0000 7fcf7bfff640 20 client.4182.objecter dump_active .. 0 homeless 2024-03-25T18:16:36.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.665+0000 7fcf7bfff640 1 -- 172.21.15.122:0/552939038 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcf8c0505a0 con 0x7fcf941430c0 2024-03-25T18:16:36.667 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.665+0000 7fcf7bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:36.667 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:36.665+0000 7fcf7bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:37.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.032+0000 7fcf99c4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:37.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.032+0000 7fcf99c4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:37.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.032+0000 7fcf99c4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:16:37.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.032+0000 7fcf99c4a640 1 -- 172.21.15.122:0/552939038 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcf940abb10 con 0x7fcf941430c0 2024-03-25T18:16:37.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.099+0000 7fcf7bfff640 1 -- 172.21.15.122:0/552939038 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fcf8c0524d0 con 0x7fcf941430c0 2024-03-25T18:16:37.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.099+0000 7fcf7bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:37.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.099+0000 7fcf7bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:37.100 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:16:37.101 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:37.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.101+0000 7fcf99c4a640 10 librados: watch_flush enter 2024-03-25T18:16:37.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.102+0000 7fcf99c4a640 10 librados: watch_flush exit 2024-03-25T18:16:37.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.102+0000 7fcf99c4a640 20 client.4182.objecter shutdown clearing up homeless session... 2024-03-25T18:16:37.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.102+0000 7fcf99c4a640 10 client.4182.objecter successfully canceled tick 2024-03-25T18:16:37.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.102+0000 7fcf99c4a640 1 -- 172.21.15.122:0/552939038 >> v1:172.21.15.204:6800/676228005 conn(0x7fcf8006dac0 legacy=0x7fcf8006ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:37.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.102+0000 7fcf99c4a640 10 monclient: shutdown 2024-03-25T18:16:37.104 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.102+0000 7fcf99c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:37.104 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.102+0000 7fcf99c4a640 1 -- 172.21.15.122:0/552939038 >> v1:172.21.15.122:6789/0 conn(0x7fcf941430c0 legacy=0x7fcf941d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:37.104 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.102+0000 7fcf99c4a640 1 -- 172.21.15.122:0/552939038 shutdown_connections 2024-03-25T18:16:37.104 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.102+0000 7fcf99c4a640 1 -- 172.21.15.122:0/552939038 >> 172.21.15.122:0/552939038 conn(0x7fcf940b7ea0 msgr2=0x7fcf9413c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:37.104 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.102+0000 7fcf99c4a640 1 -- 172.21.15.122:0/552939038 shutdown_connections 2024-03-25T18:16:37.104 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.103+0000 7fcf99c4a640 1 -- 172.21.15.122:0/552939038 wait complete. 2024-03-25T18:16:37.104 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:37.103+0000 7fcf99c4a640 1 librados: shutdown 2024-03-25T18:16:37.117 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:41.120 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:00:06.824 35673== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.130 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:00:06.829 35675== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.184 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:00:06.888 35673== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.216 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:00:06.915 35675== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.303 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:00:07.001 35676== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.330 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:00:07.034 35674== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.362 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:00:07.060 35676== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.389 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:00:07.093 35674== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.458 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:00:07.138 35508== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.517 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:00:07.198 35508== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.623 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:00:07.315 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.626 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:00:07.325 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.674 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:00:07.362 35507== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.683 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:00:07.375 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.688 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:00:07.387 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.732 INFO:tasks.ceph.osd.4.smithi204.stderr:2024-03-25T18:16:41.725+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:16:41.735 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:00:07.422 35507== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:41.779 INFO:tasks.ceph.osd.6.smithi204.stderr:2024-03-25T18:16:41.773+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:16:41.940 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-03-25T18:16:41.934+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:16:41.947 INFO:tasks.ceph.osd.5.smithi204.stderr:2024-03-25T18:16:41.938+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:16:42.080 INFO:tasks.ceph.osd.3.smithi122.stderr:2024-03-25T18:16:42.073+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:16:42.191 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:00:07.894 35673== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.237 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:00:07.936 35675== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.258 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-03-25T18:16:42.251+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:16:42.267 INFO:tasks.ceph.osd.1.smithi122.stderr:2024-03-25T18:16:42.260+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:16:42.301 INFO:tasks.ceph.osd.0.smithi122.stderr:2024-03-25T18:16:42.294+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:16:42.401 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:00:08.099 35676== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.409 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:00:08.113 35674== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.547 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:00:08.227 35508== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.554 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:00:08.258 35673== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.609 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:00:08.308 35675== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.749 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:00:08.448 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.764 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:00:08.452 35507== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.765 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:00:08.463 35676== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.766 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:00:08.458 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.785 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:00:08.489 35674== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.909 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:00:08.589 35508== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.912 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:00:08.616 35673== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:42.918 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:00:08.617 35675== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.115 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:00:08.815 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.121 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:00:08.819 35676== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.145 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:00:08.833 35507== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.147 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:00:08.851 35674== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.155 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:00:08.847 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.192 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:00:08.891 35675== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.212 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:00:08.916 35673== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.223 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:00:08.903 35508== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.409 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:00:09.107 35676== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.438 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:00:09.142 35674== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.470 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:00:09.169 35675== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.471 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:00:09.170 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.478 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:00:09.169 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.490 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:00:09.171 35508== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.505 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:00:09.209 35673== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.515 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:00:09.202 35507== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.701 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:00:09.399 35676== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.739 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:00:09.443 35674== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.743 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:00:09.442 35675== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.747 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:00:09.439 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.763 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:00:09.444 35508== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.772 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:00:09.471 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.792 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:00:09.496 35673== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.813 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:00:09.500 35507== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:43.923 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:43.966 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:00:09.664 35676== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.011+0000 7f4122785640 10 monclient: get_monmap_and_config 2024-03-25T18:16:44.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.011+0000 7f4122785640 10 monclient: build_initial_monmap 2024-03-25T18:16:44.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.011+0000 7f4122785640 10 monclient: monmap: 2024-03-25T18:16:44.012 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:16:44.012 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:44.012 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:16:44.012280+0000 2024-03-25T18:16:44.012 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:16:44.012280+0000 2024-03-25T18:16:44.012 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:44.012 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:44.012 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:44.013 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:16:44.013 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:16:44.013 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:44.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.011+0000 7f4122785640 1 Processor -- start 2024-03-25T18:16:44.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.011+0000 7f4122785640 1 -- start start 2024-03-25T18:16:44.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.011+0000 7f4122785640 10 monclient: init 2024-03-25T18:16:44.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.012+0000 7f4122785640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:44.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.012+0000 7f4122785640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:44.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.012+0000 7f4122785640 10 monclient(hunting): picked mon.noname-a con 0x7f411c0a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:44.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.012+0000 7f4122785640 10 monclient(hunting): picked mon.noname-b con 0x7f411c13f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:44.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.012+0000 7f4122785640 10 monclient(hunting): picked mon.noname-c con 0x7f411c1430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:44.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.012+0000 7f4122785640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c142520 con 0x7f411c0a98f0 2024-03-25T18:16:44.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.012+0000 7f4122785640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c0b28a0 con 0x7f411c1430c0 2024-03-25T18:16:44.015 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.012+0000 7f4122785640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c0b2a30 con 0x7f411c13f710 2024-03-25T18:16:44.015 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.012+0000 7f4122785640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:44.015 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.012+0000 7f4122785640 10 monclient(hunting): authenticate will time out at 7327.278522s 2024-03-25T18:16:44.015 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.013+0000 7f411b7fe640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f411c13f710 0x7f411c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:42192/0 (socket says 172.21.15.122:42192) 2024-03-25T18:16:44.015 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.013+0000 7f411b7fe640 1 -- 172.21.15.122:0/4021587160 learned_addr learned my addr 172.21.15.122:0/4021587160 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:16:44.015 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:00:09.715 35675== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 1 -- 172.21.15.122:0/4021587160 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090409738 0 0) 0x7f411c0b2a30 con 0x7f411c13f710 2024-03-25T18:16:44.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:44.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:44.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:44.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 10 monclient(hunting): my global_id is 4193 2024-03-25T18:16:44.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 1 -- 172.21.15.122:0/4021587160 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4104003520 con 0x7f411c13f710 2024-03-25T18:16:44.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 1 -- 172.21.15.122:0/4021587160 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307277229 0 0) 0x7f411c142520 con 0x7f411c0a98f0 2024-03-25T18:16:44.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:44.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:44.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:44.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:16:44.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 1 -- 172.21.15.122:0/4021587160 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411c0b2a30 con 0x7f411c0a98f0 2024-03-25T18:16:44.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 1 -- 172.21.15.122:0/4021587160 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102455644 0 0) 0x7f411c0b28a0 con 0x7f411c1430c0 2024-03-25T18:16:44.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:44.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:44.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:44.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:16:44.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.015+0000 7f411affd640 1 -- 172.21.15.122:0/4021587160 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411c142520 con 0x7f411c1430c0 2024-03-25T18:16:44.019 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f411affd640 1 -- 172.21.15.122:0/4021587160 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 19250188 0 0) 0x7f4104003520 con 0x7f411c13f710 2024-03-25T18:16:44.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f411affd640 1 -- 172.21.15.122:0/4021587160 >> v1:172.21.15.204:6789/0 conn(0x7f411c1430c0 legacy=0x7f411c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:44.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f411affd640 1 -- 172.21.15.122:0/4021587160 >> v1:172.21.15.122:6789/0 conn(0x7f411c0a98f0 legacy=0x7f411c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:44.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f411affd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:44.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f411affd640 1 monclient: found mon.noname-b 2024-03-25T18:16:44.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f411affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:44.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f411affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:16:44.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f411affd640 1 -- 172.21.15.122:0/4021587160 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411c0b44c0 con 0x7f411c13f710 2024-03-25T18:16:44.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f411affd640 10 monclient: _finish_auth 0 2024-03-25T18:16:44.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f411affd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:44.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f411affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:44.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f411affd640 1 -- 172.21.15.122:0/4021587160 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f410c001690 con 0x7f411c13f710 2024-03-25T18:16:44.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f4122785640 5 monclient: authenticate success, global_id 4193 2024-03-25T18:16:44.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f4122785640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:44.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f411affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:44.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f411affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:44.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f411affd640 10 monclient: dump: 2024-03-25T18:16:44.021 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:44.021 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:44.021 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:44.021 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:44.021 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:44.021 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:44.021 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:44.021 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:44.021 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:44.021 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:44.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.018+0000 7f4122785640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:44.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.020+0000 7f411affd640 1 -- 172.21.15.122:0/4021587160 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f410c0017f0 con 0x7f411c13f710 2024-03-25T18:16:44.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.020+0000 7f411affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:44.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.020+0000 7f411affd640 1 -- 172.21.15.122:0/4021587160 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f410c001c00 con 0x7f411c13f710 2024-03-25T18:16:44.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.020+0000 7f4122785640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:44.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.020+0000 7f4122785640 4 set_mon_vals no callback set 2024-03-25T18:16:44.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.020+0000 7f411affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:44.023 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.020+0000 7f411affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:44.023 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.020+0000 7f411affd640 10 monclient: dump: 2024-03-25T18:16:44.023 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:44.023 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:44.023 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:44.023 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:44.023 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:44.023 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:44.023 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:44.023 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:44.023 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:44.023 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:44.024 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:00:09.713 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.021+0000 7f4122785640 10 monclient: shutdown 2024-03-25T18:16:44.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.021+0000 7f4122785640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:44.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.021+0000 7f4122785640 1 -- 172.21.15.122:0/4021587160 >> v1:172.21.15.122:6790/0 conn(0x7f411c13f710 legacy=0x7f411c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:44.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.021+0000 7f4122785640 1 -- 172.21.15.122:0/4021587160 shutdown_connections 2024-03-25T18:16:44.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.021+0000 7f4122785640 1 -- 172.21.15.122:0/4021587160 >> 172.21.15.122:0/4021587160 conn(0x7f411c0b7ea0 msgr2=0x7f411c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:44.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.021+0000 7f4122785640 1 -- 172.21.15.122:0/4021587160 shutdown_connections 2024-03-25T18:16:44.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.021+0000 7f4122785640 1 -- 172.21.15.122:0/4021587160 wait complete. 2024-03-25T18:16:44.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.021+0000 7f4122785640 10 monclient: build_initial_monmap 2024-03-25T18:16:44.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.021+0000 7f4122785640 10 monclient: monmap: 2024-03-25T18:16:44.027 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:16:44.027 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:44.028 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:16:44.028 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:16:44.028 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:44.028 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:16:44.028 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:44.028 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:16:44.028 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:16:44.028 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:44.028 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.022+0000 7f4122785640 1 librados: starting msgr at 2024-03-25T18:16:44.028 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.022+0000 7f4122785640 1 librados: starting objecter 2024-03-25T18:16:44.028 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.022+0000 7f4122785640 1 Processor -- start 2024-03-25T18:16:44.028 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.022+0000 7f4122785640 1 -- start start 2024-03-25T18:16:44.028 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.022+0000 7f4122785640 1 librados: setting wanted keys 2024-03-25T18:16:44.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.022+0000 7f4122785640 1 librados: calling monclient init 2024-03-25T18:16:44.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.022+0000 7f4122785640 10 monclient: init 2024-03-25T18:16:44.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.022+0000 7f4122785640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:44.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.022+0000 7f4122785640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:44.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.022+0000 7f4122785640 10 monclient(hunting): picked mon.noname-a con 0x7f411c0a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:44.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.022+0000 7f4122785640 10 monclient(hunting): picked mon.noname-b con 0x7f411c13f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:44.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.022+0000 7f4122785640 10 monclient(hunting): picked mon.noname-c con 0x7f411c1430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:44.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.022+0000 7f4122785640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c0b1870 con 0x7f411c0a98f0 2024-03-25T18:16:44.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.023+0000 7f4122785640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c0ac910 con 0x7f411c13f710 2024-03-25T18:16:44.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.023+0000 7f4122785640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f411c0abb00 con 0x7f411c1430c0 2024-03-25T18:16:44.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.023+0000 7f4122785640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:44.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.023+0000 7f4122785640 10 monclient(hunting): authenticate will time out at 7327.288992s 2024-03-25T18:16:44.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.023+0000 7f4120d22640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f411c1430c0 0x7f411c1e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:42200/0 (socket says 172.21.15.122:42200) 2024-03-25T18:16:44.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.023+0000 7f4120d22640 1 -- 172.21.15.122:0/3222068749 learned_addr learned my addr 172.21.15.122:0/3222068749 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:16:44.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.023+0000 7f4118ff9640 10 client.?.objecter ms_handle_connect 0x7f411c1430c0 2024-03-25T18:16:44.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.023+0000 7f4118ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:44.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.023+0000 7f4118ff9640 10 client.?.objecter ms_handle_connect 0x7f411c0a98f0 2024-03-25T18:16:44.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.023+0000 7f4118ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:44.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.024+0000 7f4118ff9640 10 client.?.objecter ms_handle_connect 0x7f411c13f710 2024-03-25T18:16:44.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.024+0000 7f4118ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:44.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.025+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2479944566 0 0) 0x7f411c0abb00 con 0x7f411c1430c0 2024-03-25T18:16:44.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.025+0000 7f4118ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:44.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.025+0000 7f4118ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:44.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.025+0000 7f4118ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:44.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.025+0000 7f4118ff9640 10 monclient(hunting): my global_id is 4196 2024-03-25T18:16:44.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.025+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4108004eb0 con 0x7f411c1430c0 2024-03-25T18:16:44.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.025+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946204906 0 0) 0x7f411c0b2a30 con 0x7f411c0a98f0 2024-03-25T18:16:44.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.025+0000 7f4118ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:44.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.025+0000 7f4118ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:44.031 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.025+0000 7f4118ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:44.031 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.025+0000 7f4118ff9640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:16:44.031 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.025+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411c0abb00 con 0x7f411c0a98f0 2024-03-25T18:16:44.031 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.026+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3707442517 0 0) 0x7f411c0ac910 con 0x7f411c13f710 2024-03-25T18:16:44.031 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.026+0000 7f4118ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:16:44.031 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.026+0000 7f4118ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:44.031 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.026+0000 7f4118ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:44.031 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.026+0000 7f4118ff9640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:16:44.031 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.026+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f411c0b2a30 con 0x7f411c13f710 2024-03-25T18:16:44.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1935305216 0 0) 0x7f4108004eb0 con 0x7f411c1430c0 2024-03-25T18:16:44.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f411c0ac910 con 0x7f411c1430c0 2024-03-25T18:16:44.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4114001c70 con 0x7f411c1430c0 2024-03-25T18:16:44.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:44.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:44.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:44.033 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:44.033 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:44.033 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:44.033 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:44.033 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:44.033 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:44.033 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:44.033 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:44.033 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:44.033 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:44.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:44.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2282770998 0 0) 0x7f411c0b2a30 con 0x7f411c13f710 2024-03-25T18:16:44.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4108004eb0 con 0x7f411c13f710 2024-03-25T18:16:44.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f410c001b00 con 0x7f411c13f710 2024-03-25T18:16:44.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:44.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:44.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:44.034 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:44.034 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:44.034 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:44.034 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:44.034 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:44.034 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:44.034 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:44.034 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:44.035 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:44.035 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:44.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 132251921 0 0) 0x7f411c0abb00 con 0x7f411c0a98f0 2024-03-25T18:16:44.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f411c0b2a30 con 0x7f411c0a98f0 2024-03-25T18:16:44.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f41100017d0 con 0x7f411c0a98f0 2024-03-25T18:16:44.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:44.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:44.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.030+0000 7f4118ff9640 10 monclient(hunting): dump: 2024-03-25T18:16:44.035 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:44.035 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:44.035 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:44.035 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:44.035 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:44.035 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:44.035 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:44.036 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:44.036 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:44.036 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:44.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 891844439 0 0) 0x7f411c0ac910 con 0x7f411c1430c0 2024-03-25T18:16:44.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 >> v1:172.21.15.204:6789/0 conn(0x7f411c13f710 legacy=0x7f411c0ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:44.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 >> v1:172.21.15.122:6789/0 conn(0x7f411c0a98f0 legacy=0x7f411c138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:44.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4118ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:16:44.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4118ff9640 1 monclient: found mon.c 2024-03-25T18:16:44.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4118ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:44.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4118ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:16:44.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411c0abde0 con 0x7f411c1430c0 2024-03-25T18:16:44.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4118ff9640 10 monclient: _finish_auth 0 2024-03-25T18:16:44.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4118ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:16:44.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4118ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:44.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4122785640 5 monclient: authenticate success, global_id 4196 2024-03-25T18:16:44.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4122785640 10 monclient: _renew_subs 2024-03-25T18:16:44.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4122785640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:16:44.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4122785640 1 -- 172.21.15.122:0/3222068749 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f411c138f40 con 0x7f411c1430c0 2024-03-25T18:16:44.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4122785640 10 client.4196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:44.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4122785640 10 monclient: _renew_subs 2024-03-25T18:16:44.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4122785640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:16:44.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4122785640 1 -- 172.21.15.122:0/3222068749 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f411c1363f0 con 0x7f411c1430c0 2024-03-25T18:16:44.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.033+0000 7f4122785640 1 librados: init done 2024-03-25T18:16:44.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.034+0000 7f4122785640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:44.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.034+0000 7f4122785640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:44.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.034+0000 7f4122785640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:16:44.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.034+0000 7f4122785640 1 -- 172.21.15.122:0/3222068749 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f411c0627b0 con 0x7f411c1430c0 2024-03-25T18:16:44.039 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:00:09.717 35508== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.039 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:00:09.741 35674== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.049 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.048+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4114003990 con 0x7f411c1430c0 2024-03-25T18:16:44.050 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.048+0000 7f4118ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:44.050 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.048+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4114003da0 con 0x7f411c1430c0 2024-03-25T18:16:44.050 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.048+0000 7f4118ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:44.050 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.049+0000 7f4118ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:44.050 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.049+0000 7f4118ff9640 10 monclient: dump: 2024-03-25T18:16:44.050 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:44.050 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:44.050 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:44.050 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:44.050 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:44.050 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:44.050 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:44.050 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:44.051 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:44.051 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:44.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.048+0000 7f411a7fc640 4 set_mon_vals no callback set 2024-03-25T18:16:44.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.049+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f4114003fc0 con 0x7f411c1430c0 2024-03-25T18:16:44.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.050+0000 7f411bfff640 10 monclient: get_auth_request con 0x7f410806e110 auth_method 0 2024-03-25T18:16:44.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.050+0000 7f4118ff9640 10 client.4196.objecter ms_handle_connect 0x7f410806e110 2024-03-25T18:16:44.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.052+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f4114054ac0 con 0x7f411c1430c0 2024-03-25T18:16:44.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.052+0000 7f4118ff9640 10 client.4196.objecter ms_dispatch 0x7f411c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:44.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.052+0000 7f4118ff9640 3 client.4196.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:44.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.052+0000 7f4118ff9640 3 client.4196.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:44.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.052+0000 7f4118ff9640 20 client.4196.objecter dump_active .. 0 homeless 2024-03-25T18:16:44.071 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:00:09.770 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.083 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:00:09.787 35673== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.104 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:00:09.792 35507== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.254 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:00:09.952 35676== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.283 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.282+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f41140505d0 con 0x7f411c1430c0 2024-03-25T18:16:44.283 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.282+0000 7f4118ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:44.283 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.282+0000 7f4118ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:44.286 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:00:09.985 35675== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.290 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:00:09.982 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.313 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:00:09.994 35508== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.328 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:00:10.032 35674== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.367 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:00:10.066 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.371 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:00:10.075 35673== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.372 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:00:10.059 35507== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.470+0000 7f4122785640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:44.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.470+0000 7f4122785640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:44.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.470+0000 7f4122785640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:16:44.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.470+0000 7f4122785640 1 -- 172.21.15.122:0/3222068749 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f411c1434a0 con 0x7f411c1430c0 2024-03-25T18:16:44.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.515+0000 7f4118ff9640 1 -- 172.21.15.122:0/3222068749 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f4114052500 con 0x7f411c1430c0 2024-03-25T18:16:44.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.515+0000 7f4118ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:44.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.515+0000 7f4118ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:44.516 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:16:44.516 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:44.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.517+0000 7f4122785640 10 librados: watch_flush enter 2024-03-25T18:16:44.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.517+0000 7f4122785640 10 librados: watch_flush exit 2024-03-25T18:16:44.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.517+0000 7f4122785640 20 client.4196.objecter shutdown clearing up homeless session... 2024-03-25T18:16:44.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.517+0000 7f4122785640 10 client.4196.objecter successfully canceled tick 2024-03-25T18:16:44.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.517+0000 7f4122785640 1 -- 172.21.15.122:0/3222068749 >> v1:172.21.15.204:6800/676228005 conn(0x7f410806e110 legacy=0x7f41080705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:44.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.517+0000 7f4122785640 10 monclient: shutdown 2024-03-25T18:16:44.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.517+0000 7f4122785640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:44.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.517+0000 7f4122785640 1 -- 172.21.15.122:0/3222068749 >> v1:172.21.15.122:6790/0 conn(0x7f411c1430c0 legacy=0x7f411c1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:44.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.517+0000 7f4122785640 1 -- 172.21.15.122:0/3222068749 shutdown_connections 2024-03-25T18:16:44.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.517+0000 7f4122785640 1 -- 172.21.15.122:0/3222068749 >> 172.21.15.122:0/3222068749 conn(0x7f411c0b7ea0 msgr2=0x7f411c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:44.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.517+0000 7f4122785640 1 -- 172.21.15.122:0/3222068749 shutdown_connections 2024-03-25T18:16:44.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.517+0000 7f4122785640 1 -- 172.21.15.122:0/3222068749 wait complete. 2024-03-25T18:16:44.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:44.517+0000 7f4122785640 1 librados: shutdown 2024-03-25T18:16:44.528 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:44.542 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:00:10.240 35676== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.558 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:00:10.257 35675== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.570 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:00:10.262 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.590 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:00:10.271 35508== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.617 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:00:10.321 35674== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.645 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:00:10.349 35673== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.664 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:00:10.363 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.667 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:00:10.355 35507== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.836 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:00:10.528 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.841 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:00:10.539 35676== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.863 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:00:10.544 35508== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.915 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:00:10.619 35674== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.955 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:00:10.654 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:44.959 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:00:10.647 35507== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.050 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:00:10.750 35675== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.100 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:00:10.792 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.104 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:00:10.804 35675== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.175 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:00:10.879 35673== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.244 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:00:10.948 35673== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.246 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:00:10.945 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.250 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:00:10.937 35507== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.331 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:00:11.011 35508== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.385 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:00:11.066 35508== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.472 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:00:11.176 35674== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.479 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:00:11.177 35676== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.527 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:00:11.231 35674== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.534 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:00:11.232 35676== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.572 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:00:11.264 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.627 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:00:11.318 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.790 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:00:11.489 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.794 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:00:11.482 35507== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.848 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:00:11.536 35507== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:45.870 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:00:11.569 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:50.414 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:00:16.113 35675== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:50.711 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:00:16.391 35508== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:50.809 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:00:16.513 35673== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:51.102 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:00:16.793 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:51.154 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:00:16.858 35674== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:51.280 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:00:16.978 35676== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:51.333 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:51.378 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:00:17.077 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:51.457 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.456+0000 7fb0eb777640 10 monclient: get_monmap_and_config 2024-03-25T18:16:51.457 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.456+0000 7fb0eb777640 10 monclient: build_initial_monmap 2024-03-25T18:16:51.457 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.456+0000 7fb0eb777640 10 monclient: monmap: 2024-03-25T18:16:51.457 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:16:51.458 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:51.458 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:16:51.457765+0000 2024-03-25T18:16:51.458 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:16:51.457765+0000 2024-03-25T18:16:51.458 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:51.458 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:51.458 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:51.458 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:16:51.458 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:16:51.458 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:51.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.457+0000 7fb0eb777640 1 Processor -- start 2024-03-25T18:16:51.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.457+0000 7fb0eb777640 1 -- start start 2024-03-25T18:16:51.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.457+0000 7fb0eb777640 10 monclient: init 2024-03-25T18:16:51.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.458+0000 7fb0eb777640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:51.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.458+0000 7fb0eb777640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:51.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.458+0000 7fb0eb777640 10 monclient(hunting): picked mon.noname-a con 0x7fb0e40b4d60 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:51.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.458+0000 7fb0eb777640 10 monclient(hunting): picked mon.noname-c con 0x7fb0e40b1730 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:51.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.458+0000 7fb0eb777640 10 monclient(hunting): picked mon.noname-b con 0x7fb0e414be90 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:51.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.458+0000 7fb0eb777640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e40af940 con 0x7fb0e40b4d60 2024-03-25T18:16:51.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.458+0000 7fb0eb777640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e40b0a70 con 0x7fb0e40b1730 2024-03-25T18:16:51.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.458+0000 7fb0eb777640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e40b0d50 con 0x7fb0e414be90 2024-03-25T18:16:51.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.458+0000 7fb0eb777640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:51.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.458+0000 7fb0eb777640 10 monclient(hunting): authenticate will time out at 7334.724743s 2024-03-25T18:16:51.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.459+0000 7fb0e9d14640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7fb0e414be90 0x7fb0e414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:50122/0 (socket says 172.21.15.122:50122) 2024-03-25T18:16:51.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.459+0000 7fb0e9d14640 1 -- 172.21.15.122:0/2222326870 learned_addr learned my addr 172.21.15.122:0/2222326870 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:16:51.463 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.461+0000 7fb0dbfff640 1 -- 172.21.15.122:0/2222326870 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 90856101 0 0) 0x7fb0e40b0a70 con 0x7fb0e40b1730 2024-03-25T18:16:51.463 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.461+0000 7fb0dbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:51.463 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.461+0000 7fb0dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:51.463 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.461+0000 7fb0dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:51.463 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.461+0000 7fb0dbfff640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:16:51.463 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.461+0000 7fb0dbfff640 1 -- 172.21.15.122:0/2222326870 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d0003520 con 0x7fb0e40b1730 2024-03-25T18:16:51.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.462+0000 7fb0dbfff640 1 -- 172.21.15.122:0/2222326870 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3368865013 0 0) 0x7fb0e40b0d50 con 0x7fb0e414be90 2024-03-25T18:16:51.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.462+0000 7fb0dbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:51.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.462+0000 7fb0dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:51.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.462+0000 7fb0dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:51.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.462+0000 7fb0dbfff640 10 monclient(hunting): my global_id is 4199 2024-03-25T18:16:51.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.462+0000 7fb0dbfff640 1 -- 172.21.15.122:0/2222326870 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e40b0a70 con 0x7fb0e414be90 2024-03-25T18:16:51.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.463+0000 7fb0dbfff640 1 -- 172.21.15.122:0/2222326870 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 972256781 0 0) 0x7fb0e40af940 con 0x7fb0e40b4d60 2024-03-25T18:16:51.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.463+0000 7fb0dbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:51.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.463+0000 7fb0dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:51.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.463+0000 7fb0dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:51.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.463+0000 7fb0dbfff640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:16:51.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.463+0000 7fb0dbfff640 1 -- 172.21.15.122:0/2222326870 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e40b0d50 con 0x7fb0e40b4d60 2024-03-25T18:16:51.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.465+0000 7fb0dbfff640 1 -- 172.21.15.122:0/2222326870 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1044345240 0 0) 0x7fb0d0003520 con 0x7fb0e40b1730 2024-03-25T18:16:51.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.465+0000 7fb0dbfff640 1 -- 172.21.15.122:0/2222326870 >> v1:172.21.15.122:6790/0 conn(0x7fb0e414be90 legacy=0x7fb0e414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:51.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.465+0000 7fb0dbfff640 1 -- 172.21.15.122:0/2222326870 >> v1:172.21.15.122:6789/0 conn(0x7fb0e40b4d60 legacy=0x7fb0e413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:51.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.465+0000 7fb0dbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:51.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.465+0000 7fb0dbfff640 1 monclient: found mon.noname-c 2024-03-25T18:16:51.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.465+0000 7fb0dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:51.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.465+0000 7fb0dbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:16:51.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.465+0000 7fb0dbfff640 1 -- 172.21.15.122:0/2222326870 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e40b1030 con 0x7fb0e40b1730 2024-03-25T18:16:51.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.466+0000 7fb0dbfff640 10 monclient: _finish_auth 0 2024-03-25T18:16:51.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.466+0000 7fb0dbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:51.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.466+0000 7fb0dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:51.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.466+0000 7fb0dbfff640 1 -- 172.21.15.122:0/2222326870 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fb0d4001690 con 0x7fb0e40b1730 2024-03-25T18:16:51.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.466+0000 7fb0dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:51.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.466+0000 7fb0dbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:51.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.466+0000 7fb0dbfff640 10 monclient: dump: 2024-03-25T18:16:51.469 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:51.469 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:51.469 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:51.469 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:51.469 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:51.469 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:51.469 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:51.469 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:51.469 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:51.469 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:51.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.466+0000 7fb0eb777640 5 monclient: authenticate success, global_id 4213 2024-03-25T18:16:51.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.466+0000 7fb0eb777640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:51.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.469+0000 7fb0dbfff640 1 -- 172.21.15.122:0/2222326870 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0d40017f0 con 0x7fb0e40b1730 2024-03-25T18:16:51.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.469+0000 7fb0dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:51.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.469+0000 7fb0dbfff640 1 -- 172.21.15.122:0/2222326870 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fb0d4001c00 con 0x7fb0e40b1730 2024-03-25T18:16:51.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.469+0000 7fb0dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:51.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.469+0000 7fb0dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:51.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.469+0000 7fb0dbfff640 10 monclient: dump: 2024-03-25T18:16:51.471 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:51.472 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:51.472 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:51.472 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:51.472 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:51.472 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:51.472 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:51.472 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:51.472 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:51.472 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:51.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.469+0000 7fb0eb777640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:51.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.469+0000 7fb0eb777640 4 set_mon_vals no callback set 2024-03-25T18:16:51.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.470+0000 7fb0eb777640 10 monclient: shutdown 2024-03-25T18:16:51.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.470+0000 7fb0eb777640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:51.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.470+0000 7fb0eb777640 1 -- 172.21.15.122:0/2222326870 >> v1:172.21.15.204:6789/0 conn(0x7fb0e40b1730 legacy=0x7fb0e40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:51.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.470+0000 7fb0eb777640 1 -- 172.21.15.122:0/2222326870 shutdown_connections 2024-03-25T18:16:51.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.470+0000 7fb0eb777640 1 -- 172.21.15.122:0/2222326870 >> 172.21.15.122:0/2222326870 conn(0x7fb0e40617a0 msgr2=0x7fb0e4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:51.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.470+0000 7fb0eb777640 1 -- 172.21.15.122:0/2222326870 shutdown_connections 2024-03-25T18:16:51.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.470+0000 7fb0eb777640 1 -- 172.21.15.122:0/2222326870 wait complete. 2024-03-25T18:16:51.474 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.471+0000 7fb0eb777640 10 monclient: build_initial_monmap 2024-03-25T18:16:51.474 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.471+0000 7fb0eb777640 10 monclient: monmap: 2024-03-25T18:16:51.474 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:16:51.474 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:51.474 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:16:51.474 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:16:51.474 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:51.475 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:16:51.475 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:51.475 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:16:51.475 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:16:51.475 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:51.475 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.471+0000 7fb0eb777640 1 librados: starting msgr at 2024-03-25T18:16:51.475 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.471+0000 7fb0eb777640 1 librados: starting objecter 2024-03-25T18:16:51.475 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.472+0000 7fb0eb777640 1 Processor -- start 2024-03-25T18:16:51.475 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.472+0000 7fb0eb777640 1 -- start start 2024-03-25T18:16:51.475 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.472+0000 7fb0eb777640 1 librados: setting wanted keys 2024-03-25T18:16:51.475 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.472+0000 7fb0eb777640 1 librados: calling monclient init 2024-03-25T18:16:51.475 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.472+0000 7fb0eb777640 10 monclient: init 2024-03-25T18:16:51.475 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.472+0000 7fb0eb777640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:51.475 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.472+0000 7fb0eb777640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:51.475 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.472+0000 7fb0eb777640 10 monclient(hunting): picked mon.noname-c con 0x7fb0e40b1730 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:51.475 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.473+0000 7fb0eb777640 10 monclient(hunting): picked mon.noname-b con 0x7fb0e40b4d60 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:51.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.473+0000 7fb0eb777640 10 monclient(hunting): picked mon.noname-a con 0x7fb0e414be90 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:51.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.473+0000 7fb0eb777640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e41e0010 con 0x7fb0e414be90 2024-03-25T18:16:51.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.473+0000 7fb0eb777640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e413bbd0 con 0x7fb0e40b4d60 2024-03-25T18:16:51.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.473+0000 7fb0eb777640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e41dfca0 con 0x7fb0e40b1730 2024-03-25T18:16:51.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.473+0000 7fb0eb777640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:51.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.473+0000 7fb0eb777640 10 monclient(hunting): authenticate will time out at 7334.739131s 2024-03-25T18:16:51.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.473+0000 7fb0e9513640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7fb0e40b1730 0x7fb0e41df590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:50132/0 (socket says 172.21.15.122:50132) 2024-03-25T18:16:51.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.473+0000 7fb0e9513640 1 -- 172.21.15.122:0/2829033074 learned_addr learned my addr 172.21.15.122:0/2829033074 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:16:51.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.473+0000 7fb0d9ffb640 10 client.?.objecter ms_handle_connect 0x7fb0e40b1730 2024-03-25T18:16:51.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.473+0000 7fb0d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:51.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.474+0000 7fb0d9ffb640 10 client.?.objecter ms_handle_connect 0x7fb0e40b4d60 2024-03-25T18:16:51.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.474+0000 7fb0d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:51.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.474+0000 7fb0d9ffb640 10 client.?.objecter ms_handle_connect 0x7fb0e414be90 2024-03-25T18:16:51.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.474+0000 7fb0d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:51.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.476+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338140027 0 0) 0x7fb0e41dfca0 con 0x7fb0e40b1730 2024-03-25T18:16:51.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.476+0000 7fb0d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:51.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.476+0000 7fb0d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:51.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.476+0000 7fb0d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:51.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.476+0000 7fb0d9ffb640 10 monclient(hunting): my global_id is 4202 2024-03-25T18:16:51.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.476+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0c8004eb0 con 0x7fb0e40b1730 2024-03-25T18:16:51.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.477+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903041659 0 0) 0x7fb0e413bbd0 con 0x7fb0e40b4d60 2024-03-25T18:16:51.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.477+0000 7fb0d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:51.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.477+0000 7fb0d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:51.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.477+0000 7fb0d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:51.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.477+0000 7fb0d9ffb640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:16:51.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.477+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e41dfca0 con 0x7fb0e40b4d60 2024-03-25T18:16:51.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.477+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110607105 0 0) 0x7fb0e41e0010 con 0x7fb0e414be90 2024-03-25T18:16:51.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.477+0000 7fb0d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:51.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.477+0000 7fb0d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:51.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.477+0000 7fb0d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:51.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.477+0000 7fb0d9ffb640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:16:51.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.477+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e413bbd0 con 0x7fb0e414be90 2024-03-25T18:16:51.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.482+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3679131221 0 0) 0x7fb0e41dfca0 con 0x7fb0e40b4d60 2024-03-25T18:16:51.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.482+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0e41e0010 con 0x7fb0e40b4d60 2024-03-25T18:16:51.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.482+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fb0d4001b00 con 0x7fb0e40b4d60 2024-03-25T18:16:51.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.482+0000 7fb0d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:51.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.482+0000 7fb0d9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:51.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.482+0000 7fb0d9ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:51.483 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:51.483 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:51.484 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:51.484 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:51.484 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:51.484 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:51.484 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:51.484 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:51.484 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:51.484 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:51.484 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.482+0000 7fb0d9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:51.486 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.484+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2745443008 0 0) 0x7fb0c8004eb0 con 0x7fb0e40b1730 2024-03-25T18:16:51.486 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.484+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0e41dfca0 con 0x7fb0e40b1730 2024-03-25T18:16:51.486 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.484+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fb0e0001a10 con 0x7fb0e40b1730 2024-03-25T18:16:51.487 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.484+0000 7fb0d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:51.487 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.484+0000 7fb0d9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:51.487 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.484+0000 7fb0d9ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:51.487 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:51.487 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:51.487 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:51.487 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:51.487 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:51.487 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:51.487 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:51.487 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:51.487 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:51.487 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:51.487 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.485+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2775701588 0 0) 0x7fb0e413bbd0 con 0x7fb0e414be90 2024-03-25T18:16:51.487 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.485+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0c8004eb0 con 0x7fb0e414be90 2024-03-25T18:16:51.487 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.485+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fb0dc001bd0 con 0x7fb0e414be90 2024-03-25T18:16:51.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.485+0000 7fb0d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:51.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.485+0000 7fb0d9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:51.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.485+0000 7fb0d9ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:51.488 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:51.488 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:51.488 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:51.488 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:51.488 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:51.488 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:51.488 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:51.488 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:51.488 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:51.488 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:51.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.486+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3862614284 0 0) 0x7fb0e41e0010 con 0x7fb0e40b4d60 2024-03-25T18:16:51.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.486+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 >> v1:172.21.15.122:6790/0 conn(0x7fb0e40b1730 legacy=0x7fb0e41df590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:51.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.486+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 >> v1:172.21.15.122:6789/0 conn(0x7fb0e414be90 legacy=0x7fb0e4141aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:51.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.486+0000 7fb0d9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:51.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.486+0000 7fb0d9ffb640 1 monclient: found mon.b 2024-03-25T18:16:51.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.486+0000 7fb0d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:51.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.486+0000 7fb0d9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:51.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.486+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e413b2f0 con 0x7fb0e40b4d60 2024-03-25T18:16:51.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.486+0000 7fb0d9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:51.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.486+0000 7fb0d9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:51.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.486+0000 7fb0d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:51.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.486+0000 7fb0eb777640 5 monclient: authenticate success, global_id 4216 2024-03-25T18:16:51.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.486+0000 7fb0eb777640 10 monclient: _renew_subs 2024-03-25T18:16:51.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.486+0000 7fb0eb777640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:51.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.486+0000 7fb0eb777640 1 -- 172.21.15.122:0/2829033074 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e413b5a0 con 0x7fb0e40b4d60 2024-03-25T18:16:51.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.487+0000 7fb0eb777640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:51.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.487+0000 7fb0eb777640 10 monclient: _renew_subs 2024-03-25T18:16:51.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.487+0000 7fb0eb777640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:51.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.487+0000 7fb0eb777640 1 -- 172.21.15.122:0/2829033074 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e41dec90 con 0x7fb0e40b4d60 2024-03-25T18:16:51.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.487+0000 7fb0eb777640 1 librados: init done 2024-03-25T18:16:51.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.488+0000 7fb0eb777640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:51.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.488+0000 7fb0eb777640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:51.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.488+0000 7fb0eb777640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:51.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.488+0000 7fb0eb777640 1 -- 172.21.15.122:0/2829033074 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e40afbd0 con 0x7fb0e40b4d60 2024-03-25T18:16:51.591 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:00:17.278 35507== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:51.747 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.746+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0d4001ca0 con 0x7fb0e40b4d60 2024-03-25T18:16:51.747 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.746+0000 7fb0d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:51.747 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.746+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fb0d4003a40 con 0x7fb0e40b4d60 2024-03-25T18:16:51.747 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.746+0000 7fb0d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:51.747 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.746+0000 7fb0d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:51.747 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.746+0000 7fb0d9ffb640 10 monclient: dump: 2024-03-25T18:16:51.748 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:51.748 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:51.748 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:51.748 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:51.748 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:51.748 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:51.748 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:51.748 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:51.748 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:51.748 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:51.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.746+0000 7fb0db7fe640 4 set_mon_vals no callback set 2024-03-25T18:16:51.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.747+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fb0d4019a10 con 0x7fb0e40b4d60 2024-03-25T18:16:51.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.748+0000 7fb0e9513640 10 monclient: get_auth_request con 0x7fb0c806e0f0 auth_method 0 2024-03-25T18:16:51.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.748+0000 7fb0d9ffb640 10 client.4216.objecter ms_handle_connect 0x7fb0c806e0f0 2024-03-25T18:16:51.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.753+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fb0d40535a0 con 0x7fb0e40b4d60 2024-03-25T18:16:51.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.753+0000 7fb0d9ffb640 10 client.4216.objecter ms_dispatch 0x7fb0e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:51.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.753+0000 7fb0d9ffb640 3 client.4216.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:51.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.753+0000 7fb0d9ffb640 3 client.4216.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:51.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.753+0000 7fb0d9ffb640 20 client.4216.objecter dump_active .. 0 homeless 2024-03-25T18:16:51.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.754+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb0d404eee0 con 0x7fb0e40b4d60 2024-03-25T18:16:51.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.754+0000 7fb0d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:51.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:51.754+0000 7fb0d9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:52.049 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.047+0000 7fb0eb777640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:52.049 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.047+0000 7fb0eb777640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:52.049 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.047+0000 7fb0eb777640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:52.049 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.047+0000 7fb0eb777640 1 -- 172.21.15.122:0/2829033074 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0e41dfe30 con 0x7fb0e40b4d60 2024-03-25T18:16:52.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.073+0000 7fb0d9ffb640 1 -- 172.21.15.122:0/2829033074 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fb0d4001210 con 0x7fb0e40b4d60 2024-03-25T18:16:52.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.073+0000 7fb0d9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:52.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.073+0000 7fb0d9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:52.074 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:16:52.074 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:52.076 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.075+0000 7fb0eb777640 10 librados: watch_flush enter 2024-03-25T18:16:52.076 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.075+0000 7fb0eb777640 10 librados: watch_flush exit 2024-03-25T18:16:52.076 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.075+0000 7fb0eb777640 20 client.4216.objecter shutdown clearing up homeless session... 2024-03-25T18:16:52.076 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.075+0000 7fb0eb777640 10 client.4216.objecter successfully canceled tick 2024-03-25T18:16:52.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.075+0000 7fb0eb777640 1 -- 172.21.15.122:0/2829033074 >> v1:172.21.15.204:6800/676228005 conn(0x7fb0c806e0f0 legacy=0x7fb0c80705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:52.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.075+0000 7fb0eb777640 10 monclient: shutdown 2024-03-25T18:16:52.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.075+0000 7fb0eb777640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:52.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.075+0000 7fb0eb777640 1 -- 172.21.15.122:0/2829033074 >> v1:172.21.15.204:6789/0 conn(0x7fb0e40b4d60 legacy=0x7fb0e41de3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:52.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.075+0000 7fb0eb777640 1 -- 172.21.15.122:0/2829033074 shutdown_connections 2024-03-25T18:16:52.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.075+0000 7fb0eb777640 1 -- 172.21.15.122:0/2829033074 >> 172.21.15.122:0/2829033074 conn(0x7fb0e40617a0 msgr2=0x7fb0e413c610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:52.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.075+0000 7fb0eb777640 1 -- 172.21.15.122:0/2829033074 shutdown_connections 2024-03-25T18:16:52.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.075+0000 7fb0eb777640 1 -- 172.21.15.122:0/2829033074 wait complete. 2024-03-25T18:16:52.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:52.075+0000 7fb0eb777640 1 librados: shutdown 2024-03-25T18:16:52.086 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:56.172 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:00:21.871 35675== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:56.178 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:00:21.878 35675== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:56.196 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:00:21.877 35508== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:56.202 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:00:21.883 35508== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:56.313 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:00:22.017 35673== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:56.317 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:00:22.021 35673== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:56.547 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:00:22.239 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:56.551 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:00:22.243 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:57.044 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:00:22.748 35674== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:57.049 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:00:22.748 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:57.050 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:00:22.754 35674== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:57.055 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:00:22.754 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:57.289 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:00:22.987 35676== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:57.295 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:00:22.993 35676== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:57.625 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:00:23.313 35507== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:57.631 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:00:23.319 35507== Warning: unimplemented fcntl command: 1038 2024-03-25T18:16:58.891 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:59.002 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.001+0000 7f7b7399d640 10 monclient: get_monmap_and_config 2024-03-25T18:16:59.002 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.001+0000 7f7b7399d640 10 monclient: build_initial_monmap 2024-03-25T18:16:59.002 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.001+0000 7f7b7399d640 10 monclient: monmap: 2024-03-25T18:16:59.002 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:16:59.002 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:59.002 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:16:59.002196+0000 2024-03-25T18:16:59.003 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:16:59.002196+0000 2024-03-25T18:16:59.003 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:59.003 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:59.003 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:59.003 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:16:59.003 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:16:59.003 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:59.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.001+0000 7f7b7399d640 1 Processor -- start 2024-03-25T18:16:59.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.001+0000 7f7b7399d640 1 -- start start 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.001+0000 7f7b7399d640 10 monclient: init 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.002+0000 7f7b7399d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.002+0000 7f7b7399d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.002+0000 7f7b7399d640 10 monclient(hunting): picked mon.noname-c con 0x7f7b6c0b4d60 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.002+0000 7f7b7399d640 10 monclient(hunting): picked mon.noname-b con 0x7f7b6c0b1730 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.002+0000 7f7b7399d640 10 monclient(hunting): picked mon.noname-a con 0x7f7b6c14be90 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.002+0000 7f7b7399d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c0af940 con 0x7f7b6c14be90 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.002+0000 7f7b7399d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c0b0a70 con 0x7f7b6c0b4d60 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.002+0000 7f7b7399d640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c0b0d50 con 0x7f7b6c0b1730 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.002+0000 7f7b7399d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.002+0000 7f7b7399d640 10 monclient(hunting): authenticate will time out at 7342.268781s 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.003+0000 7f7b70f38640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f7b6c0b1730 0x7f7b6c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:50148/0 (socket says 172.21.15.122:50148) 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.003+0000 7f7b70f38640 1 -- 172.21.15.122:0/1629508303 learned_addr learned my addr 172.21.15.122:0/1629508303 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:16:59.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 1 -- 172.21.15.122:0/1629508303 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737538114 0 0) 0x7f7b6c0b0a70 con 0x7f7b6c0b4d60 2024-03-25T18:16:59.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:59.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:59.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:59.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 10 monclient(hunting): my global_id is 4219 2024-03-25T18:16:59.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 1 -- 172.21.15.122:0/1629508303 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b58003520 con 0x7f7b6c0b4d60 2024-03-25T18:16:59.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 1 -- 172.21.15.122:0/1629508303 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341542713 0 0) 0x7f7b6c0b0d50 con 0x7f7b6c0b1730 2024-03-25T18:16:59.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:59.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:59.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:59.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 10 monclient(hunting): my global_id is 4205 2024-03-25T18:16:59.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 1 -- 172.21.15.122:0/1629508303 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b6c0b0a70 con 0x7f7b6c0b1730 2024-03-25T18:16:59.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 1 -- 172.21.15.122:0/1629508303 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108229945 0 0) 0x7f7b6c0af940 con 0x7f7b6c14be90 2024-03-25T18:16:59.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:59.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:59.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:59.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:16:59.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.006+0000 7f7b63fff640 1 -- 172.21.15.122:0/1629508303 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b6c0b0d50 con 0x7f7b6c14be90 2024-03-25T18:16:59.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.010+0000 7f7b63fff640 1 -- 172.21.15.122:0/1629508303 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1492828185 0 0) 0x7f7b58003520 con 0x7f7b6c0b4d60 2024-03-25T18:16:59.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.010+0000 7f7b63fff640 1 -- 172.21.15.122:0/1629508303 >> v1:172.21.15.122:6790/0 conn(0x7f7b6c0b1730 legacy=0x7f7b6c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:59.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.010+0000 7f7b63fff640 1 -- 172.21.15.122:0/1629508303 >> v1:172.21.15.122:6789/0 conn(0x7f7b6c14be90 legacy=0x7f7b6c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:59.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.010+0000 7f7b63fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:59.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.010+0000 7f7b63fff640 1 monclient: found mon.noname-c 2024-03-25T18:16:59.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.010+0000 7f7b63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:59.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.010+0000 7f7b63fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:16:59.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.010+0000 7f7b63fff640 1 -- 172.21.15.122:0/1629508303 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c0b1030 con 0x7f7b6c0b4d60 2024-03-25T18:16:59.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.010+0000 7f7b63fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:59.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.010+0000 7f7b63fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:59.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.010+0000 7f7b63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:59.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.010+0000 7f7b63fff640 1 -- 172.21.15.122:0/1629508303 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f7b680015c0 con 0x7f7b6c0b4d60 2024-03-25T18:16:59.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.010+0000 7f7b63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:59.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.010+0000 7f7b63fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:59.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.010+0000 7f7b63fff640 10 monclient: dump: 2024-03-25T18:16:59.012 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:59.012 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:59.012 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:59.012 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:59.012 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:59.012 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:59.012 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:59.013 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:59.013 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:59.013 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:59.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.010+0000 7f7b7399d640 5 monclient: authenticate success, global_id 4219 2024-03-25T18:16:59.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.010+0000 7f7b7399d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:59.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.012+0000 7f7b63fff640 1 -- 172.21.15.122:0/1629508303 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b68001720 con 0x7f7b6c0b4d60 2024-03-25T18:16:59.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.012+0000 7f7b63fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:59.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.012+0000 7f7b7399d640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:59.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.012+0000 7f7b7399d640 4 set_mon_vals no callback set 2024-03-25T18:16:59.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.012+0000 7f7b63fff640 1 -- 172.21.15.122:0/1629508303 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f7b68001b30 con 0x7f7b6c0b4d60 2024-03-25T18:16:59.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.012+0000 7f7b63fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:59.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.012+0000 7f7b63fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:59.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.012+0000 7f7b63fff640 10 monclient: dump: 2024-03-25T18:16:59.014 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:59.014 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:59.015 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:59.015 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:59.015 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:59.015 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:59.015 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:59.015 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:59.015 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:59.015 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:59.015 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.013+0000 7f7b7399d640 10 monclient: shutdown 2024-03-25T18:16:59.015 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.013+0000 7f7b7399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:59.015 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.013+0000 7f7b7399d640 1 -- 172.21.15.122:0/1629508303 >> v1:172.21.15.204:6789/0 conn(0x7f7b6c0b4d60 legacy=0x7f7b6c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:59.015 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.013+0000 7f7b7399d640 1 -- 172.21.15.122:0/1629508303 shutdown_connections 2024-03-25T18:16:59.015 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.013+0000 7f7b7399d640 1 -- 172.21.15.122:0/1629508303 >> 172.21.15.122:0/1629508303 conn(0x7f7b6c0617a0 msgr2=0x7f7b6c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:59.015 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.013+0000 7f7b7399d640 1 -- 172.21.15.122:0/1629508303 shutdown_connections 2024-03-25T18:16:59.015 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.013+0000 7f7b7399d640 1 -- 172.21.15.122:0/1629508303 wait complete. 2024-03-25T18:16:59.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.014+0000 7f7b7399d640 10 monclient: build_initial_monmap 2024-03-25T18:16:59.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.014+0000 7f7b7399d640 10 monclient: monmap: 2024-03-25T18:16:59.017 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:16:59.017 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:59.017 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:16:59.017 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:16:59.017 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:59.017 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:16:59.017 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:16:59.017 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:16:59.018 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:16:59.018 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:59.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.014+0000 7f7b7399d640 1 librados: starting msgr at 2024-03-25T18:16:59.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.014+0000 7f7b7399d640 1 librados: starting objecter 2024-03-25T18:16:59.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.014+0000 7f7b7399d640 1 Processor -- start 2024-03-25T18:16:59.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.015+0000 7f7b7399d640 1 -- start start 2024-03-25T18:16:59.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.015+0000 7f7b7399d640 1 librados: setting wanted keys 2024-03-25T18:16:59.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.015+0000 7f7b7399d640 1 librados: calling monclient init 2024-03-25T18:16:59.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.015+0000 7f7b7399d640 10 monclient: init 2024-03-25T18:16:59.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.015+0000 7f7b7399d640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:59.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.015+0000 7f7b7399d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:59.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.015+0000 7f7b7399d640 10 monclient(hunting): picked mon.noname-b con 0x7f7b6c0b1730 addr v1:172.21.15.204:6789/0 2024-03-25T18:16:59.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.015+0000 7f7b7399d640 10 monclient(hunting): picked mon.noname-c con 0x7f7b6c0b4d60 addr v1:172.21.15.122:6790/0 2024-03-25T18:16:59.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.016+0000 7f7b7399d640 10 monclient(hunting): picked mon.noname-a con 0x7f7b6c14be90 addr v1:172.21.15.122:6789/0 2024-03-25T18:16:59.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.016+0000 7f7b7399d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c1df8b0 con 0x7f7b6c14be90 2024-03-25T18:16:59.019 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.016+0000 7f7b7399d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c13c880 con 0x7f7b6c0b1730 2024-03-25T18:16:59.019 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.016+0000 7f7b70f38640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f7b6c0b4d60 0x7f7b6c13ba60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:50158/0 (socket says 172.21.15.122:50158) 2024-03-25T18:16:59.019 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.016+0000 7f7b70f38640 1 -- 172.21.15.122:0/1662506456 learned_addr learned my addr 172.21.15.122:0/1662506456 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:16:59.019 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.016+0000 7f7b7399d640 1 -- 172.21.15.122:0/1662506456 --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b6c06fae0 con 0x7f7b6c0b4d60 2024-03-25T18:16:59.019 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.016+0000 7f7b7399d640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:59.019 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.016+0000 7f7b7399d640 10 monclient(hunting): authenticate will time out at 7342.282484s 2024-03-25T18:16:59.019 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.016+0000 7f7b61ffb640 10 client.?.objecter ms_handle_connect 0x7f7b6c0b1730 2024-03-25T18:16:59.019 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.016+0000 7f7b61ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:59.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.016+0000 7f7b61ffb640 10 client.?.objecter ms_handle_connect 0x7f7b6c0b4d60 2024-03-25T18:16:59.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.016+0000 7f7b61ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:59.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.017+0000 7f7b61ffb640 10 client.?.objecter ms_handle_connect 0x7f7b6c14be90 2024-03-25T18:16:59.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.017+0000 7f7b61ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:59.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.019+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829899764 0 0) 0x7f7b6c13c880 con 0x7f7b6c0b1730 2024-03-25T18:16:59.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.019+0000 7f7b61ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:59.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.019+0000 7f7b61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:59.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.019+0000 7f7b61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:59.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.019+0000 7f7b61ffb640 10 monclient(hunting): my global_id is 4222 2024-03-25T18:16:59.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.019+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b50004eb0 con 0x7f7b6c0b1730 2024-03-25T18:16:59.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.020+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295667473 0 0) 0x7f7b6c1df8b0 con 0x7f7b6c14be90 2024-03-25T18:16:59.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.020+0000 7f7b61ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:59.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.020+0000 7f7b61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:59.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.020+0000 7f7b61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:59.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.020+0000 7f7b61ffb640 10 monclient(hunting): my global_id is 4224 2024-03-25T18:16:59.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.020+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b6c13c880 con 0x7f7b6c14be90 2024-03-25T18:16:59.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.020+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 498646268 0 0) 0x7f7b6c06fae0 con 0x7f7b6c0b4d60 2024-03-25T18:16:59.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.020+0000 7f7b61ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:59.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.020+0000 7f7b61ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:59.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.020+0000 7f7b61ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:59.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.020+0000 7f7b61ffb640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:16:59.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.020+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b6c1df8b0 con 0x7f7b6c0b4d60 2024-03-25T18:16:59.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.026+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2965254804 0 0) 0x7f7b50004eb0 con 0x7f7b6c0b1730 2024-03-25T18:16:59.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.026+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b6c06fae0 con 0x7f7b6c0b1730 2024-03-25T18:16:59.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.026+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f7b68001b30 con 0x7f7b6c0b1730 2024-03-25T18:16:59.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.026+0000 7f7b61ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:59.028 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.026+0000 7f7b61ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:59.028 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.026+0000 7f7b61ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:59.028 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:59.028 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:59.028 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:59.028 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:59.028 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:59.028 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:59.028 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:59.028 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:59.028 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:59.028 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:59.028 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.026+0000 7f7b61ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:59.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.027+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3010100499 0 0) 0x7f7b6c13c880 con 0x7f7b6c14be90 2024-03-25T18:16:59.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.027+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b50004eb0 con 0x7f7b6c14be90 2024-03-25T18:16:59.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.027+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f7b64001b50 con 0x7f7b6c14be90 2024-03-25T18:16:59.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.027+0000 7f7b61ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:59.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.027+0000 7f7b61ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:59.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.027+0000 7f7b61ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:59.030 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:59.030 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:59.030 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:59.030 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:59.030 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:59.031 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:59.031 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:59.031 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:59.031 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:59.031 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:59.031 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.027+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 590409924 0 0) 0x7f7b6c1df8b0 con 0x7f7b6c0b4d60 2024-03-25T18:16:59.031 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.027+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b6c13c880 con 0x7f7b6c0b4d60 2024-03-25T18:16:59.031 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.027+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f7b5c0018f0 con 0x7f7b6c0b4d60 2024-03-25T18:16:59.031 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.027+0000 7f7b61ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:59.031 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.027+0000 7f7b61ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:59.031 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.027+0000 7f7b61ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:59.031 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:59.031 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:59.031 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:59.032 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:59.032 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:59.032 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:59.032 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:59.032 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:59.032 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:59.032 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:59.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3771687269 0 0) 0x7f7b6c06fae0 con 0x7f7b6c0b1730 2024-03-25T18:16:59.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 >> v1:172.21.15.122:6790/0 conn(0x7f7b6c0b4d60 legacy=0x7f7b6c13ba60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:59.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 >> v1:172.21.15.122:6789/0 conn(0x7f7b6c14be90 legacy=0x7f7b6c13c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:59.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b61ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:59.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b61ffb640 1 monclient: found mon.b 2024-03-25T18:16:59.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b61ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:59.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b61ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:59.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b6c06fdc0 con 0x7f7b6c0b1730 2024-03-25T18:16:59.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b61ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:59.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b61ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:59.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b61ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:59.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b7399d640 5 monclient: authenticate success, global_id 4222 2024-03-25T18:16:59.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b7399d640 10 monclient: _renew_subs 2024-03-25T18:16:59.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b7399d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:59.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b7399d640 1 -- 172.21.15.122:0/1662506456 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b6c0700a0 con 0x7f7b6c0b1730 2024-03-25T18:16:59.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b7399d640 10 client.4222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:59.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b7399d640 10 monclient: _renew_subs 2024-03-25T18:16:59.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b7399d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:59.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b7399d640 1 -- 172.21.15.122:0/1662506456 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b6c1dff20 con 0x7f7b6c0b1730 2024-03-25T18:16:59.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.031+0000 7f7b7399d640 1 librados: init done 2024-03-25T18:16:59.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.032+0000 7f7b7399d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:59.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.032+0000 7f7b7399d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:59.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.032+0000 7f7b7399d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:59.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.032+0000 7f7b7399d640 1 -- 172.21.15.122:0/1662506456 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b6c0627b0 con 0x7f7b6c0b1730 2024-03-25T18:16:59.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.056+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b68003910 con 0x7f7b6c0b1730 2024-03-25T18:16:59.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.056+0000 7f7b61ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:59.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.056+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f7b68000f30 con 0x7f7b6c0b1730 2024-03-25T18:16:59.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.056+0000 7f7b61ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:59.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.056+0000 7f7b61ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:59.058 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.056+0000 7f7b61ffb640 10 monclient: dump: 2024-03-25T18:16:59.058 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:16:59.058 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:16:59.058 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:59.058 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:16:59.058 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:59.058 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:16:59.058 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:16:59.058 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:16:59.058 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:16:59.058 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:16:59.059 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.056+0000 7f7b637fe640 4 set_mon_vals no callback set 2024-03-25T18:16:59.059 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.058+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f7b68003e20 con 0x7f7b6c0b1730 2024-03-25T18:16:59.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.059+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f7b680537e0 con 0x7f7b6c0b1730 2024-03-25T18:16:59.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.059+0000 7f7b61ffb640 10 client.4222.objecter ms_dispatch 0x7f7b6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:59.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.059+0000 7f7b61ffb640 3 client.4222.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:59.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.059+0000 7f7b61ffb640 3 client.4222.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:59.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.059+0000 7f7b61ffb640 20 client.4222.objecter dump_active .. 0 homeless 2024-03-25T18:16:59.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.059+0000 7f7b70f38640 10 monclient: get_auth_request con 0x7f7b5006e0f0 auth_method 0 2024-03-25T18:16:59.061 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.059+0000 7f7b61ffb640 10 client.4222.objecter ms_handle_connect 0x7f7b5006e0f0 2024-03-25T18:16:59.316 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.315+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b6804f2f0 con 0x7f7b6c0b1730 2024-03-25T18:16:59.316 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.315+0000 7f7b61ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:59.316 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.315+0000 7f7b61ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:59.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.599+0000 7f7b7399d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:59.601 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.599+0000 7f7b7399d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:59.601 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.599+0000 7f7b7399d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:16:59.601 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.599+0000 7f7b7399d640 1 -- 172.21.15.122:0/1662506456 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b6c069f40 con 0x7f7b6c0b1730 2024-03-25T18:16:59.624 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.623+0000 7f7b61ffb640 1 -- 172.21.15.122:0/1662506456 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f7b68051220 con 0x7f7b6c0b1730 2024-03-25T18:16:59.624 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.623+0000 7f7b61ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:59.624 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.623+0000 7f7b61ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:59.624 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:16:59.624 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:59.626 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.625+0000 7f7b7399d640 10 librados: watch_flush enter 2024-03-25T18:16:59.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.625+0000 7f7b7399d640 10 librados: watch_flush exit 2024-03-25T18:16:59.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.625+0000 7f7b7399d640 20 client.4222.objecter shutdown clearing up homeless session... 2024-03-25T18:16:59.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.625+0000 7f7b7399d640 10 client.4222.objecter successfully canceled tick 2024-03-25T18:16:59.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.625+0000 7f7b7399d640 1 -- 172.21.15.122:0/1662506456 >> v1:172.21.15.204:6800/676228005 conn(0x7f7b5006e0f0 legacy=0x7f7b500705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:59.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.625+0000 7f7b7399d640 10 monclient: shutdown 2024-03-25T18:16:59.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.625+0000 7f7b7399d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:59.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.625+0000 7f7b7399d640 1 -- 172.21.15.122:0/1662506456 >> v1:172.21.15.204:6789/0 conn(0x7f7b6c0b1730 legacy=0x7f7b6c0704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:59.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.625+0000 7f7b7399d640 1 -- 172.21.15.122:0/1662506456 shutdown_connections 2024-03-25T18:16:59.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.625+0000 7f7b7399d640 1 -- 172.21.15.122:0/1662506456 >> 172.21.15.122:0/1662506456 conn(0x7f7b6c0617a0 msgr2=0x7f7b6c0b8540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:59.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.625+0000 7f7b7399d640 1 -- 172.21.15.122:0/1662506456 shutdown_connections 2024-03-25T18:16:59.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.625+0000 7f7b7399d640 1 -- 172.21.15.122:0/1662506456 wait complete. 2024-03-25T18:16:59.628 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:16:59.625+0000 7f7b7399d640 1 librados: shutdown 2024-03-25T18:16:59.638 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:00.301 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:00:26.001 35675== Warning: unimplemented fcntl command: 1038 2024-03-25T18:17:00.373 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:00:26.054 35508== Warning: unimplemented fcntl command: 1038 2024-03-25T18:17:00.391 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:00:26.095 35673== Warning: unimplemented fcntl command: 1038 2024-03-25T18:17:00.917 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:00:26.607 35506== Warning: unimplemented fcntl command: 1038 2024-03-25T18:17:01.055 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:00:26.759 35674== Warning: unimplemented fcntl command: 1038 2024-03-25T18:17:01.488 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:00:27.187 35505== Warning: unimplemented fcntl command: 1038 2024-03-25T18:17:01.757 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:00:27.455 35676== Warning: unimplemented fcntl command: 1038 2024-03-25T18:17:01.803 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:00:27.490 35507== Warning: unimplemented fcntl command: 1038 2024-03-25T18:17:06.444 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:06.577 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.576+0000 7fd9c659f640 10 monclient: get_monmap_and_config 2024-03-25T18:17:06.577 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.576+0000 7fd9c659f640 10 monclient: build_initial_monmap 2024-03-25T18:17:06.578 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.576+0000 7fd9c659f640 10 monclient: monmap: 2024-03-25T18:17:06.578 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:17:06.578 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:06.578 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:17:06.577834+0000 2024-03-25T18:17:06.578 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:17:06.577834+0000 2024-03-25T18:17:06.578 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:06.578 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:06.578 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:17:06.578 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:17:06.578 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:17:06.579 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:06.580 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.577+0000 7fd9c659f640 1 Processor -- start 2024-03-25T18:17:06.580 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.577+0000 7fd9c659f640 1 -- start start 2024-03-25T18:17:06.580 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.577+0000 7fd9c659f640 10 monclient: init 2024-03-25T18:17:06.580 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.578+0000 7fd9c659f640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:06.580 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.578+0000 7fd9c659f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:06.580 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.578+0000 7fd9c659f640 10 monclient(hunting): picked mon.noname-a con 0x7fd9c00b49a0 addr v1:172.21.15.122:6789/0 2024-03-25T18:17:06.580 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.578+0000 7fd9c659f640 10 monclient(hunting): picked mon.noname-c con 0x7fd9c00a9680 addr v1:172.21.15.204:6789/0 2024-03-25T18:17:06.580 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.578+0000 7fd9c659f640 10 monclient(hunting): picked mon.noname-b con 0x7fd9c00b3fa0 addr v1:172.21.15.122:6790/0 2024-03-25T18:17:06.580 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.578+0000 7fd9c659f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9c013fe70 con 0x7fd9c00b49a0 2024-03-25T18:17:06.581 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.578+0000 7fd9c659f640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9c00ab250 con 0x7fd9c00a9680 2024-03-25T18:17:06.581 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.578+0000 7fd9c659f640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9c00a9b10 con 0x7fd9c00b3fa0 2024-03-25T18:17:06.581 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.578+0000 7fd9c659f640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:06.581 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.578+0000 7fd9c659f640 10 monclient(hunting): authenticate will time out at 7349.844344s 2024-03-25T18:17:06.581 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.578+0000 7fd9c559d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd9c00b49a0 0x7fd9c013f760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:48682/0 (socket says 172.21.15.122:48682) 2024-03-25T18:17:06.581 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.578+0000 7fd9c559d640 1 -- 172.21.15.122:0/94353677 learned_addr learned my addr 172.21.15.122:0/94353677 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:17:06.582 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.581+0000 7fd9b7fff640 1 -- 172.21.15.122:0/94353677 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808079722 0 0) 0x7fd9bc001110 con 0x7fd9c00b49a0 2024-03-25T18:17:06.582 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.581+0000 7fd9b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.582 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.581+0000 7fd9b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.583 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.581+0000 7fd9b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.583 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.581+0000 7fd9b7fff640 10 monclient(hunting): my global_id is 4227 2024-03-25T18:17:06.583 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.581+0000 7fd9b7fff640 1 -- 172.21.15.122:0/94353677 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9ac003520 con 0x7fd9c00b49a0 2024-03-25T18:17:06.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.582+0000 7fd9b7fff640 1 -- 172.21.15.122:0/94353677 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2209811289 0 0) 0x7fd9c00a9b10 con 0x7fd9c00b3fa0 2024-03-25T18:17:06.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.582+0000 7fd9b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.582+0000 7fd9b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.582+0000 7fd9b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.582+0000 7fd9b7fff640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:17:06.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.582+0000 7fd9b7fff640 1 -- 172.21.15.122:0/94353677 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9bc001110 con 0x7fd9c00b3fa0 2024-03-25T18:17:06.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.582+0000 7fd9b7fff640 1 -- 172.21.15.122:0/94353677 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680752376 0 0) 0x7fd9c00ab250 con 0x7fd9c00a9680 2024-03-25T18:17:06.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.582+0000 7fd9b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.582+0000 7fd9b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.582+0000 7fd9b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.582+0000 7fd9b7fff640 10 monclient(hunting): my global_id is 4225 2024-03-25T18:17:06.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.582+0000 7fd9b7fff640 1 -- 172.21.15.122:0/94353677 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c00a9b10 con 0x7fd9c00a9680 2024-03-25T18:17:06.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.585+0000 7fd9b7fff640 1 -- 172.21.15.122:0/94353677 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2469208662 0 0) 0x7fd9ac003520 con 0x7fd9c00b49a0 2024-03-25T18:17:06.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.585+0000 7fd9b7fff640 1 -- 172.21.15.122:0/94353677 >> v1:172.21.15.122:6790/0 conn(0x7fd9c00b3fa0 legacy=0x7fd9c0145f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.585+0000 7fd9b7fff640 1 -- 172.21.15.122:0/94353677 >> v1:172.21.15.204:6789/0 conn(0x7fd9c00a9680 legacy=0x7fd9c0145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.585+0000 7fd9b7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:06.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.585+0000 7fd9b7fff640 1 monclient: found mon.noname-a 2024-03-25T18:17:06.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.585+0000 7fd9b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:06.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.585+0000 7fd9b7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:17:06.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.585+0000 7fd9b7fff640 1 -- 172.21.15.122:0/94353677 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c0151b50 con 0x7fd9c00b49a0 2024-03-25T18:17:06.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.585+0000 7fd9b7fff640 10 monclient: _finish_auth 0 2024-03-25T18:17:06.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.585+0000 7fd9b7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:06.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.585+0000 7fd9b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:06.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.585+0000 7fd9b7fff640 1 -- 172.21.15.122:0/94353677 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd9bc001680 con 0x7fd9c00b49a0 2024-03-25T18:17:06.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.585+0000 7fd9c659f640 5 monclient: authenticate success, global_id 4227 2024-03-25T18:17:06.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.585+0000 7fd9b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:06.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.585+0000 7fd9b7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:06.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.585+0000 7fd9b7fff640 10 monclient: dump: 2024-03-25T18:17:06.589 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:06.589 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:06.589 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:06.589 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:06.589 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:06.589 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:06.589 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:06.589 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:06.589 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:06.589 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:06.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.585+0000 7fd9c659f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:06.592 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.587+0000 7fd9b7fff640 1 -- 172.21.15.122:0/94353677 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9bc0017e0 con 0x7fd9c00b49a0 2024-03-25T18:17:06.592 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.587+0000 7fd9b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:06.592 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.587+0000 7fd9b7fff640 1 -- 172.21.15.122:0/94353677 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd9bc001b70 con 0x7fd9c00b49a0 2024-03-25T18:17:06.592 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.587+0000 7fd9b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:06.592 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.587+0000 7fd9b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:06.592 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.587+0000 7fd9b7fff640 10 monclient: dump: 2024-03-25T18:17:06.592 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:06.592 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:06.592 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:06.592 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:06.592 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:06.592 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:06.592 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:06.592 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:06.593 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:06.593 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:06.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.587+0000 7fd9c659f640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:06.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.587+0000 7fd9c659f640 4 set_mon_vals no callback set 2024-03-25T18:17:06.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.588+0000 7fd9c659f640 10 monclient: shutdown 2024-03-25T18:17:06.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.588+0000 7fd9c659f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:06.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.588+0000 7fd9c659f640 1 -- 172.21.15.122:0/94353677 >> v1:172.21.15.122:6789/0 conn(0x7fd9c00b49a0 legacy=0x7fd9c013f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.588+0000 7fd9c659f640 1 -- 172.21.15.122:0/94353677 shutdown_connections 2024-03-25T18:17:06.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.588+0000 7fd9c659f640 1 -- 172.21.15.122:0/94353677 >> 172.21.15.122:0/94353677 conn(0x7fd9c00615c0 msgr2=0x7fd9c00619a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:06.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.589+0000 7fd9c659f640 1 -- 172.21.15.122:0/94353677 shutdown_connections 2024-03-25T18:17:06.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.589+0000 7fd9c659f640 1 -- 172.21.15.122:0/94353677 wait complete. 2024-03-25T18:17:06.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.590+0000 7fd9c659f640 10 monclient: build_initial_monmap 2024-03-25T18:17:06.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.590+0000 7fd9c659f640 10 monclient: monmap: 2024-03-25T18:17:06.593 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:17:06.593 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:06.594 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:17:06.594 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:17:06.594 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:06.594 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:17:06.594 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:17:06.594 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:17:06.594 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:17:06.594 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:06.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.590+0000 7fd9c659f640 1 librados: starting msgr at 2024-03-25T18:17:06.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.590+0000 7fd9c659f640 1 librados: starting objecter 2024-03-25T18:17:06.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.590+0000 7fd9c659f640 1 Processor -- start 2024-03-25T18:17:06.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.591+0000 7fd9c659f640 1 -- start start 2024-03-25T18:17:06.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.591+0000 7fd9c659f640 1 librados: setting wanted keys 2024-03-25T18:17:06.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.591+0000 7fd9c659f640 1 librados: calling monclient init 2024-03-25T18:17:06.595 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.591+0000 7fd9c659f640 10 monclient: init 2024-03-25T18:17:06.597 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.591+0000 7fd9c659f640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:06.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.592+0000 7fd9c659f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:06.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.592+0000 7fd9c659f640 10 monclient(hunting): picked mon.noname-b con 0x7fd9c00b3fa0 addr v1:172.21.15.204:6789/0 2024-03-25T18:17:06.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.592+0000 7fd9c659f640 10 monclient(hunting): picked mon.noname-c con 0x7fd9c00b49a0 addr v1:172.21.15.122:6790/0 2024-03-25T18:17:06.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.592+0000 7fd9c659f640 10 monclient(hunting): picked mon.noname-a con 0x7fd9c01db4c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:17:06.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.592+0000 7fd9c659f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9c013c790 con 0x7fd9c01db4c0 2024-03-25T18:17:06.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.592+0000 7fd9c659f640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9c013aed0 con 0x7fd9c00b3fa0 2024-03-25T18:17:06.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.592+0000 7fd9c659f640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9c013c9c0 con 0x7fd9c00b49a0 2024-03-25T18:17:06.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.592+0000 7fd9c659f640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:06.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.592+0000 7fd9c659f640 10 monclient(hunting): authenticate will time out at 7349.858392s 2024-03-25T18:17:06.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.592+0000 7fd9c4d9c640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7fd9c00b49a0 0x7fd9c01dadb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:50054/0 (socket says 172.21.15.122:50054) 2024-03-25T18:17:06.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.592+0000 7fd9c4d9c640 1 -- 172.21.15.122:0/3767257064 learned_addr learned my addr 172.21.15.122:0/3767257064 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:17:06.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.592+0000 7fd9b5ffb640 10 client.?.objecter ms_handle_connect 0x7fd9c00b49a0 2024-03-25T18:17:06.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.592+0000 7fd9b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:06.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.593+0000 7fd9b5ffb640 10 client.?.objecter ms_handle_connect 0x7fd9c01db4c0 2024-03-25T18:17:06.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.593+0000 7fd9b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:06.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.593+0000 7fd9b5ffb640 10 client.?.objecter ms_handle_connect 0x7fd9c00b3fa0 2024-03-25T18:17:06.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.593+0000 7fd9b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:06.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.595+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221252132 0 0) 0x7fd9c013c9c0 con 0x7fd9c00b49a0 2024-03-25T18:17:06.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.595+0000 7fd9b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.595+0000 7fd9b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.595+0000 7fd9b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.595+0000 7fd9b5ffb640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:17:06.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.595+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9a4004eb0 con 0x7fd9c00b49a0 2024-03-25T18:17:06.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.596+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580648938 0 0) 0x7fd9c013aed0 con 0x7fd9c00b3fa0 2024-03-25T18:17:06.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.596+0000 7fd9b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.596+0000 7fd9b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.596+0000 7fd9b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.596+0000 7fd9b5ffb640 10 monclient(hunting): my global_id is 4228 2024-03-25T18:17:06.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.596+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c013c9c0 con 0x7fd9c00b3fa0 2024-03-25T18:17:06.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.596+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4092548476 0 0) 0x7fd9c013c790 con 0x7fd9c01db4c0 2024-03-25T18:17:06.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.596+0000 7fd9b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.596+0000 7fd9b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.596+0000 7fd9b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.596+0000 7fd9b5ffb640 10 monclient(hunting): my global_id is 4230 2024-03-25T18:17:06.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.596+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9c013aed0 con 0x7fd9c01db4c0 2024-03-25T18:17:06.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.603+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3268815414 0 0) 0x7fd9a4004eb0 con 0x7fd9c00b49a0 2024-03-25T18:17:06.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.603+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9c013c790 con 0x7fd9c00b49a0 2024-03-25T18:17:06.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.603+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd9b0001a10 con 0x7fd9c00b49a0 2024-03-25T18:17:06.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.603+0000 7fd9b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:06.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.603+0000 7fd9b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:06.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.603+0000 7fd9b5ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:06.605 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:06.605 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:06.605 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:06.605 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:06.605 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:06.605 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:06.605 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:06.606 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:06.606 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:06.606 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:06.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.603+0000 7fd9b5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:06.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.603+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2560848773 0 0) 0x7fd9c013c9c0 con 0x7fd9c00b3fa0 2024-03-25T18:17:06.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.603+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd9a4004eb0 con 0x7fd9c00b3fa0 2024-03-25T18:17:06.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.603+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd9bc000f90 con 0x7fd9c00b3fa0 2024-03-25T18:17:06.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.603+0000 7fd9b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:06.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.603+0000 7fd9b5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:06.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.603+0000 7fd9b5ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:06.606 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:06.606 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:06.606 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:06.606 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:06.606 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:06.607 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:06.607 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:06.607 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:06.607 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:06.607 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:06.610 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.608+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3034563897 0 0) 0x7fd9a4004eb0 con 0x7fd9c00b3fa0 2024-03-25T18:17:06.610 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.608+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 >> v1:172.21.15.122:6790/0 conn(0x7fd9c00b49a0 legacy=0x7fd9c01dadb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.608+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 >> v1:172.21.15.122:6789/0 conn(0x7fd9c01db4c0 legacy=0x7fd9c01e22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.608+0000 7fd9b5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.608+0000 7fd9b5ffb640 1 monclient: found mon.b 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.608+0000 7fd9b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.608+0000 7fd9b5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.608+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9c013c090 con 0x7fd9c00b3fa0 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.608+0000 7fd9b5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.608+0000 7fd9b5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.608+0000 7fd9b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.608+0000 7fd9c659f640 5 monclient: authenticate success, global_id 4228 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.608+0000 7fd9c659f640 10 monclient: _renew_subs 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.608+0000 7fd9c659f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.608+0000 7fd9c659f640 1 -- 172.21.15.122:0/3767257064 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9c0139d70 con 0x7fd9c00b3fa0 2024-03-25T18:17:06.612 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.609+0000 7fd9c659f640 10 client.4228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:06.612 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.609+0000 7fd9c659f640 10 monclient: _renew_subs 2024-03-25T18:17:06.612 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.609+0000 7fd9c659f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:06.612 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.609+0000 7fd9c659f640 1 -- 172.21.15.122:0/3767257064 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9c013b0e0 con 0x7fd9c00b3fa0 2024-03-25T18:17:06.612 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.609+0000 7fd9c659f640 1 librados: init done 2024-03-25T18:17:06.612 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.610+0000 7fd9c659f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:06.613 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.610+0000 7fd9c659f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:06.613 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.610+0000 7fd9c659f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:06.613 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:06.610+0000 7fd9c659f640 1 -- 172.21.15.122:0/3767257064 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9c00627b0 con 0x7fd9c00b3fa0 2024-03-25T18:17:07.047 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.046+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9bc004a90 con 0x7fd9c00b3fa0 2024-03-25T18:17:07.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.046+0000 7fd9b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:07.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.046+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd9bc000f90 con 0x7fd9c00b3fa0 2024-03-25T18:17:07.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.046+0000 7fd9b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:07.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.046+0000 7fd9b5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:07.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.046+0000 7fd9b5ffb640 10 monclient: dump: 2024-03-25T18:17:07.048 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:07.048 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:07.048 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:07.048 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:07.048 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:07.048 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:07.048 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:07.049 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:07.049 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:07.049 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:07.049 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.046+0000 7fd9b77fe640 4 set_mon_vals no callback set 2024-03-25T18:17:07.049 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.047+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fd9bc01aa70 con 0x7fd9c00b3fa0 2024-03-25T18:17:07.049 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.048+0000 7fd9c4d9c640 10 monclient: get_auth_request con 0x7fd9a406e1a0 auth_method 0 2024-03-25T18:17:07.050 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.048+0000 7fd9b5ffb640 10 client.4228.objecter ms_handle_connect 0x7fd9a406e1a0 2024-03-25T18:17:07.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.055+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fd9bc054560 con 0x7fd9c00b3fa0 2024-03-25T18:17:07.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.055+0000 7fd9b5ffb640 10 client.4228.objecter ms_dispatch 0x7fd9c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:07.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.055+0000 7fd9b5ffb640 3 client.4228.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:07.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.055+0000 7fd9b5ffb640 3 client.4228.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:07.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.055+0000 7fd9b5ffb640 20 client.4228.objecter dump_active .. 0 homeless 2024-03-25T18:17:07.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.056+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd9bc050070 con 0x7fd9c00b3fa0 2024-03-25T18:17:07.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.056+0000 7fd9b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:07.058 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.056+0000 7fd9b5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:07.310 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.308+0000 7fd9c659f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:07.310 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.308+0000 7fd9c659f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:07.310 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.308+0000 7fd9c659f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:07.310 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.308+0000 7fd9c659f640 1 -- 172.21.15.122:0/3767257064 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd9c00b16d0 con 0x7fd9c00b3fa0 2024-03-25T18:17:07.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.335+0000 7fd9b5ffb640 1 -- 172.21.15.122:0/3767257064 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fd9bc051fa0 con 0x7fd9c00b3fa0 2024-03-25T18:17:07.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.335+0000 7fd9b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:07.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.335+0000 7fd9b5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:07.336 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:17:07.337 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:07.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.338+0000 7fd9c659f640 10 librados: watch_flush enter 2024-03-25T18:17:07.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.338+0000 7fd9c659f640 10 librados: watch_flush exit 2024-03-25T18:17:07.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.338+0000 7fd9c659f640 20 client.4228.objecter shutdown clearing up homeless session... 2024-03-25T18:17:07.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.338+0000 7fd9c659f640 10 client.4228.objecter successfully canceled tick 2024-03-25T18:17:07.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.338+0000 7fd9c659f640 1 -- 172.21.15.122:0/3767257064 >> v1:172.21.15.204:6800/676228005 conn(0x7fd9a406e1a0 legacy=0x7fd9a4070660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:07.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.338+0000 7fd9c659f640 10 monclient: shutdown 2024-03-25T18:17:07.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.338+0000 7fd9c659f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:07.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.338+0000 7fd9c659f640 1 -- 172.21.15.122:0/3767257064 >> v1:172.21.15.204:6789/0 conn(0x7fd9c00b3fa0 legacy=0x7fd9c013a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:07.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.338+0000 7fd9c659f640 1 -- 172.21.15.122:0/3767257064 shutdown_connections 2024-03-25T18:17:07.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.338+0000 7fd9c659f640 1 -- 172.21.15.122:0/3767257064 >> 172.21.15.122:0/3767257064 conn(0x7fd9c00615c0 msgr2=0x7fd9c00b5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:07.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.338+0000 7fd9c659f640 1 -- 172.21.15.122:0/3767257064 shutdown_connections 2024-03-25T18:17:07.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.338+0000 7fd9c659f640 1 -- 172.21.15.122:0/3767257064 wait complete. 2024-03-25T18:17:07.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:07.338+0000 7fd9c659f640 1 librados: shutdown 2024-03-25T18:17:07.353 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:08.871 INFO:tasks.ceph.osd.6.smithi204.stderr:2024-03-25T18:17:08.866+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:17:09.087 INFO:tasks.ceph.osd.3.smithi122.stderr:2024-03-25T18:17:09.078+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:17:09.161 INFO:tasks.ceph.osd.4.smithi204.stderr:2024-03-25T18:17:09.156+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:17:09.631 INFO:tasks.ceph.osd.1.smithi122.stderr:2024-03-25T18:17:09.627+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:17:09.911 INFO:tasks.ceph.osd.5.smithi204.stderr:2024-03-25T18:17:09.907+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:17:09.975 INFO:tasks.ceph.osd.2.smithi122.stderr:2024-03-25T18:17:09.971+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:17:10.384 INFO:tasks.ceph.osd.7.smithi204.stderr:2024-03-25T18:17:10.379+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:17:10.726 INFO:tasks.ceph.osd.0.smithi122.stderr:2024-03-25T18:17:10.723+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:17:14.160 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:14.276 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.275+0000 7fc7f3796640 10 monclient: get_monmap_and_config 2024-03-25T18:17:14.276 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.275+0000 7fc7f3796640 10 monclient: build_initial_monmap 2024-03-25T18:17:14.276 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.275+0000 7fc7f3796640 10 monclient: monmap: 2024-03-25T18:17:14.276 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:17:14.276 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:14.276 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:17:14.275881+0000 2024-03-25T18:17:14.276 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:17:14.275881+0000 2024-03-25T18:17:14.276 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:14.276 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:14.276 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:17:14.276 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:17:14.276 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:17:14.277 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:14.277 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.276+0000 7fc7f3796640 1 Processor -- start 2024-03-25T18:17:14.277 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.277+0000 7fc7f3796640 1 -- start start 2024-03-25T18:17:14.277 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.277+0000 7fc7f3796640 10 monclient: init 2024-03-25T18:17:14.278 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.278+0000 7fc7f3796640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:14.278 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.278+0000 7fc7f3796640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:14.278 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.278+0000 7fc7f3796640 10 monclient(hunting): picked mon.noname-a con 0x7fc7ec0ae6e0 addr v1:172.21.15.122:6789/0 2024-03-25T18:17:14.278 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.278+0000 7fc7f3796640 10 monclient(hunting): picked mon.noname-c con 0x7fc7ec06f080 addr v1:172.21.15.204:6789/0 2024-03-25T18:17:14.278 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.278+0000 7fc7f3796640 10 monclient(hunting): picked mon.noname-b con 0x7fc7ec0704e0 addr v1:172.21.15.122:6790/0 2024-03-25T18:17:14.278 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.278+0000 7fc7f3796640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ec0718d0 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.278 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.278+0000 7fc7f3796640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ec06fa30 con 0x7fc7ec06f080 2024-03-25T18:17:14.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.278+0000 7fc7f3796640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ec070090 con 0x7fc7ec0704e0 2024-03-25T18:17:14.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.278+0000 7fc7f3796640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:14.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.278+0000 7fc7f3796640 10 monclient(hunting): authenticate will time out at 7357.543178s 2024-03-25T18:17:14.284 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.284+0000 7fc7f1532640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc7ec0ae6e0 0x7fc7ec13c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:60476/0 (socket says 172.21.15.122:60476) 2024-03-25T18:17:14.284 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.284+0000 7fc7f1532640 1 -- 172.21.15.122:0/732331513 learned_addr learned my addr 172.21.15.122:0/732331513 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:17:14.296 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.296+0000 7fc7e3fff640 1 -- 172.21.15.122:0/732331513 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036226067 0 0) 0x7fc7ec0718d0 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.297 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.296+0000 7fc7e3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:14.297 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.296+0000 7fc7e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:14.297 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.296+0000 7fc7e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:14.297 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.296+0000 7fc7e3fff640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:17:14.297 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.297+0000 7fc7e3fff640 1 -- 172.21.15.122:0/732331513 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7d8003520 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.302+0000 7fc7e3fff640 1 -- 172.21.15.122:0/732331513 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729586618 0 0) 0x7fc7ec070090 con 0x7fc7ec0704e0 2024-03-25T18:17:14.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.302+0000 7fc7e3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:14.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.302+0000 7fc7e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:14.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.302+0000 7fc7e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:14.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.302+0000 7fc7e3fff640 10 monclient(hunting): my global_id is 4241 2024-03-25T18:17:14.303 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.302+0000 7fc7e3fff640 1 -- 172.21.15.122:0/732331513 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7ec0718d0 con 0x7fc7ec0704e0 2024-03-25T18:17:14.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7e3fff640 1 -- 172.21.15.122:0/732331513 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4263225248 0 0) 0x7fc7d8003520 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7e3fff640 1 -- 172.21.15.122:0/732331513 >> v1:172.21.15.122:6790/0 conn(0x7fc7ec0704e0 legacy=0x7fc7ec14a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7e3fff640 1 -- 172.21.15.122:0/732331513 >> v1:172.21.15.204:6789/0 conn(0x7fc7ec06f080 legacy=0x7fc7ec071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7e3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:14.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7e3fff640 1 monclient: found mon.noname-a 2024-03-25T18:17:14.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:14.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7e3fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:17:14.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7e3fff640 1 -- 172.21.15.122:0/732331513 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ec06f510 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7e3fff640 10 monclient: _finish_auth 0 2024-03-25T18:17:14.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7e3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:14.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:14.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7e3fff640 1 -- 172.21.15.122:0/732331513 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fc7dc001690 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7f3796640 5 monclient: authenticate success, global_id 4257 2024-03-25T18:17:14.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7f3796640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:14.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:14.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7e3fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:14.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7e3fff640 10 monclient: dump: 2024-03-25T18:17:14.313 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:14.313 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:14.313 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:14.313 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:14.314 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:14.314 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:14.314 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:14.314 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:14.314 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:14.314 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:14.314 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.312+0000 7fc7f3796640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:14.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.327+0000 7fc7e3fff640 1 -- 172.21.15.122:0/732331513 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7dc0017f0 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.327+0000 7fc7e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.327+0000 7fc7e3fff640 1 -- 172.21.15.122:0/732331513 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fc7dc001c00 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.327+0000 7fc7e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.327+0000 7fc7e3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.327+0000 7fc7e3fff640 10 monclient: dump: 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.327+0000 7fc7f3796640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:14.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.327+0000 7fc7f3796640 4 set_mon_vals no callback set 2024-03-25T18:17:14.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.328+0000 7fc7f3796640 10 monclient: shutdown 2024-03-25T18:17:14.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.328+0000 7fc7f3796640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:14.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.328+0000 7fc7f3796640 1 -- 172.21.15.122:0/732331513 >> v1:172.21.15.122:6789/0 conn(0x7fc7ec0ae6e0 legacy=0x7fc7ec13c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.328+0000 7fc7f3796640 1 -- 172.21.15.122:0/732331513 shutdown_connections 2024-03-25T18:17:14.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.328+0000 7fc7f3796640 1 -- 172.21.15.122:0/732331513 >> 172.21.15.122:0/732331513 conn(0x7fc7ec06b630 msgr2=0x7fc7ec0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:14.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.329+0000 7fc7f3796640 1 -- 172.21.15.122:0/732331513 shutdown_connections 2024-03-25T18:17:14.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.329+0000 7fc7f3796640 1 -- 172.21.15.122:0/732331513 wait complete. 2024-03-25T18:17:14.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.329+0000 7fc7f3796640 10 monclient: build_initial_monmap 2024-03-25T18:17:14.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.329+0000 7fc7f3796640 10 monclient: monmap: 2024-03-25T18:17:14.331 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:17:14.331 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:14.331 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:17:14.331 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:17:14.331 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:14.331 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:17:14.331 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:17:14.331 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:17:14.331 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:17:14.331 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:14.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.330+0000 7fc7f3796640 1 librados: starting msgr at 2024-03-25T18:17:14.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.330+0000 7fc7f3796640 1 librados: starting objecter 2024-03-25T18:17:14.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.330+0000 7fc7f3796640 1 Processor -- start 2024-03-25T18:17:14.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.330+0000 7fc7f3796640 1 -- start start 2024-03-25T18:17:14.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.330+0000 7fc7f3796640 1 librados: setting wanted keys 2024-03-25T18:17:14.332 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.330+0000 7fc7f3796640 1 librados: calling monclient init 2024-03-25T18:17:14.332 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.330+0000 7fc7f3796640 10 monclient: init 2024-03-25T18:17:14.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.330+0000 7fc7f3796640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:14.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.330+0000 7fc7f3796640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:14.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.330+0000 7fc7f3796640 10 monclient(hunting): picked mon.noname-c con 0x7fc7ec0704e0 addr v1:172.21.15.122:6790/0 2024-03-25T18:17:14.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.331+0000 7fc7f3796640 10 monclient(hunting): picked mon.noname-a con 0x7fc7ec0ae6e0 addr v1:172.21.15.122:6789/0 2024-03-25T18:17:14.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.331+0000 7fc7f3796640 10 monclient(hunting): picked mon.noname-b con 0x7fc7ec14c030 addr v1:172.21.15.204:6789/0 2024-03-25T18:17:14.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.331+0000 7fc7f3796640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ec14c790 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.331+0000 7fc7f3796640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ec14c4a0 con 0x7fc7ec14c030 2024-03-25T18:17:14.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.331+0000 7fc7f3796640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7ec14ddd0 con 0x7fc7ec0704e0 2024-03-25T18:17:14.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.331+0000 7fc7f3796640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:14.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.331+0000 7fc7f3796640 10 monclient(hunting): authenticate will time out at 7357.596054s 2024-03-25T18:17:14.339 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.340+0000 7fc7f1532640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7fc7ec0704e0 0x7fc7ec13ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:39800/0 (socket says 172.21.15.122:39800) 2024-03-25T18:17:14.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.340+0000 7fc7f1532640 1 -- 172.21.15.122:0/831179043 learned_addr learned my addr 172.21.15.122:0/831179043 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:17:14.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.341+0000 7fc7e1ffb640 10 client.?.objecter ms_handle_connect 0x7fc7ec0704e0 2024-03-25T18:17:14.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.341+0000 7fc7e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:14.342 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.342+0000 7fc7e1ffb640 10 client.?.objecter ms_handle_connect 0x7fc7ec0ae6e0 2024-03-25T18:17:14.342 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.342+0000 7fc7e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:14.346 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.346+0000 7fc7e1ffb640 10 client.?.objecter ms_handle_connect 0x7fc7ec14c030 2024-03-25T18:17:14.346 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.346+0000 7fc7e1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:14.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.356+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689901976 0 0) 0x7fc7ec14ddd0 con 0x7fc7ec0704e0 2024-03-25T18:17:14.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.356+0000 7fc7e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:14.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.356+0000 7fc7e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:14.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.356+0000 7fc7e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:14.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.356+0000 7fc7e1ffb640 10 monclient(hunting): my global_id is 4244 2024-03-25T18:17:14.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.356+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7d0004eb0 con 0x7fc7ec0704e0 2024-03-25T18:17:14.357 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.357+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665377642 0 0) 0x7fc7ec14c4a0 con 0x7fc7ec14c030 2024-03-25T18:17:14.357 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.357+0000 7fc7e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:14.357 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.357+0000 7fc7e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:14.357 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.357+0000 7fc7e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:14.357 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.357+0000 7fc7e1ffb640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:17:14.357 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.357+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7ec14ddd0 con 0x7fc7ec14c030 2024-03-25T18:17:14.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.357+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956478355 0 0) 0x7fc7ec14c790 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.358+0000 7fc7e1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:14.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.358+0000 7fc7e1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:14.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.358+0000 7fc7e1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:14.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.358+0000 7fc7e1ffb640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:17:14.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.358+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7ec14c4a0 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.373 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.373+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2707807580 0 0) 0x7fc7ec14c4a0 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.373 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.373+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7ec14c790 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.373 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.373+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fc7e8001830 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.373 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.373+0000 7fc7e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:14.373 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.373+0000 7fc7e1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:14.373 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.373+0000 7fc7e1ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:14.373 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:14.373 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:14.374 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:14.374 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:14.374 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:14.374 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:14.374 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:14.374 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:14.374 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:14.374 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:14.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.373+0000 7fc7e1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:14.375 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.375+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 905214897 0 0) 0x7fc7d0004eb0 con 0x7fc7ec0704e0 2024-03-25T18:17:14.376 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.375+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7ec14c4a0 con 0x7fc7ec0704e0 2024-03-25T18:17:14.376 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.375+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fc7dc001bc0 con 0x7fc7ec0704e0 2024-03-25T18:17:14.376 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.375+0000 7fc7e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:14.376 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.375+0000 7fc7e1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:14.376 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.375+0000 7fc7e1ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:14.376 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:14.376 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:14.376 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:14.376 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:14.376 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:14.376 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:14.376 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:14.376 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:14.377 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:14.377 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:14.379 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.379+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4022192736 0 0) 0x7fc7ec14ddd0 con 0x7fc7ec14c030 2024-03-25T18:17:14.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.379+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc7d0004eb0 con 0x7fc7ec14c030 2024-03-25T18:17:14.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.379+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fc7e4001cb0 con 0x7fc7ec14c030 2024-03-25T18:17:14.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.379+0000 7fc7e1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:14.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.379+0000 7fc7e1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:14.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.379+0000 7fc7e1ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:14.380 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:14.380 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:14.380 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:14.380 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:14.380 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:14.380 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:14.380 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:14.380 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:14.381 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:14.381 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:14.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3476529754 0 0) 0x7fc7ec14c790 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 >> v1:172.21.15.122:6790/0 conn(0x7fc7ec0704e0 legacy=0x7fc7ec13ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 >> v1:172.21.15.204:6789/0 conn(0x7fc7ec14c030 legacy=0x7fc7ec1e2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7e1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:14.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7e1ffb640 1 monclient: found mon.a 2024-03-25T18:17:14.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7e1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:14.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7e1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:17:14.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7ec140aa0 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7e1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:14.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7e1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:14.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7e1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:14.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7f3796640 5 monclient: authenticate success, global_id 4260 2024-03-25T18:17:14.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7f3796640 10 monclient: _renew_subs 2024-03-25T18:17:14.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7f3796640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:17:14.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7f3796640 1 -- 172.21.15.122:0/831179043 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7ec14cc20 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7f3796640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:14.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7f3796640 10 monclient: _renew_subs 2024-03-25T18:17:14.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7f3796640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:17:14.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7f3796640 1 -- 172.21.15.122:0/831179043 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7ec1e3b50 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.385+0000 7fc7f3796640 1 librados: init done 2024-03-25T18:17:14.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.386+0000 7fc7f3796640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:14.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.386+0000 7fc7f3796640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:14.388 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.386+0000 7fc7f3796640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:17:14.388 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.386+0000 7fc7f3796640 1 -- 172.21.15.122:0/831179043 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7ec0627b0 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.726 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.726+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7e8001d40 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.727 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.726+0000 7fc7e1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:14.727 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.726+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fc7e8003790 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.727 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.726+0000 7fc7e1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:14.727 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.726+0000 7fc7e1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:14.727 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.726+0000 7fc7e1ffb640 10 monclient: dump: 2024-03-25T18:17:14.727 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:14.727 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:14.727 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:14.727 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:14.727 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:14.727 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:14.727 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:14.727 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:14.728 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:14.728 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:14.728 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.726+0000 7fc7e37fe640 4 set_mon_vals no callback set 2024-03-25T18:17:14.728 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.727+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fc7e80039b0 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.728 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.727+0000 7fc7f1532640 10 monclient: get_auth_request con 0x7fc7d006e0d0 auth_method 0 2024-03-25T18:17:14.728 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.728+0000 7fc7e1ffb640 10 client.4260.objecter ms_handle_connect 0x7fc7d006e0d0 2024-03-25T18:17:14.733 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.732+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fc7e8053550 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.733 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.732+0000 7fc7e1ffb640 10 client.4260.objecter ms_dispatch 0x7fc7ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:14.733 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.732+0000 7fc7e1ffb640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:14.733 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.732+0000 7fc7e1ffb640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:14.733 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.733+0000 7fc7e1ffb640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-25T18:17:14.733 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.733+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc7e804ee90 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.733 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.733+0000 7fc7e1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:14.733 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.733+0000 7fc7e1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:14.963 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.963+0000 7fc7f3796640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:14.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.963+0000 7fc7f3796640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:14.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.963+0000 7fc7f3796640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:17:14.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.963+0000 7fc7f3796640 1 -- 172.21.15.122:0/831179043 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7ec0b3430 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.991 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.990+0000 7fc7e1ffb640 1 -- 172.21.15.122:0/831179043 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fc7e8050f90 con 0x7fc7ec0ae6e0 2024-03-25T18:17:14.991 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.990+0000 7fc7e1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:14.991 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.990+0000 7fc7e1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:14.991 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:17:14.991 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:14.993 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.992+0000 7fc7f3796640 10 librados: watch_flush enter 2024-03-25T18:17:14.993 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.992+0000 7fc7f3796640 10 librados: watch_flush exit 2024-03-25T18:17:14.993 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.992+0000 7fc7f3796640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-25T18:17:14.993 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.992+0000 7fc7f3796640 10 client.4260.objecter successfully canceled tick 2024-03-25T18:17:14.993 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.993+0000 7fc7f3796640 1 -- 172.21.15.122:0/831179043 >> v1:172.21.15.204:6800/676228005 conn(0x7fc7d006e0d0 legacy=0x7fc7d0070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.994 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.993+0000 7fc7f3796640 10 monclient: shutdown 2024-03-25T18:17:14.994 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.993+0000 7fc7f3796640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:14.994 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.993+0000 7fc7f3796640 1 -- 172.21.15.122:0/831179043 >> v1:172.21.15.122:6789/0 conn(0x7fc7ec0ae6e0 legacy=0x7fc7ec140390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.994 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.993+0000 7fc7f3796640 1 -- 172.21.15.122:0/831179043 shutdown_connections 2024-03-25T18:17:14.994 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.993+0000 7fc7f3796640 1 -- 172.21.15.122:0/831179043 >> 172.21.15.122:0/831179043 conn(0x7fc7ec06b630 msgr2=0x7fc7ec13c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:14.994 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.993+0000 7fc7f3796640 1 -- 172.21.15.122:0/831179043 shutdown_connections 2024-03-25T18:17:14.994 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.993+0000 7fc7f3796640 1 -- 172.21.15.122:0/831179043 wait complete. 2024-03-25T18:17:14.994 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:14.993+0000 7fc7f3796640 1 librados: shutdown 2024-03-25T18:17:15.004 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:21.811 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:21.929 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.928+0000 7f302ea9c640 10 monclient: get_monmap_and_config 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.928+0000 7f302ea9c640 10 monclient: build_initial_monmap 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.928+0000 7f302ea9c640 10 monclient: monmap: 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:17:21.930019+0000 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:17:21.930019+0000 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:21.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.929+0000 7f302ea9c640 1 Processor -- start 2024-03-25T18:17:21.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.929+0000 7f302ea9c640 1 -- start start 2024-03-25T18:17:21.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.929+0000 7f302ea9c640 10 monclient: init 2024-03-25T18:17:21.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.930+0000 7f302ea9c640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:21.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.930+0000 7f302ea9c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:21.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.930+0000 7f302ea9c640 10 monclient(hunting): picked mon.noname-a con 0x7f30280a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:17:21.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.930+0000 7f302ea9c640 10 monclient(hunting): picked mon.noname-c con 0x7f302813f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:17:21.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.930+0000 7f302ea9c640 10 monclient(hunting): picked mon.noname-b con 0x7f30281430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:17:21.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.930+0000 7f302ea9c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3028142520 con 0x7f30280a98f0 2024-03-25T18:17:21.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.930+0000 7f302ea9c640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30280b28a0 con 0x7f302813f710 2024-03-25T18:17:21.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.930+0000 7f302ea9c640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30280b2a30 con 0x7f30281430c0 2024-03-25T18:17:21.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.930+0000 7f302ea9c640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:21.933 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.930+0000 7f302ea9c640 10 monclient(hunting): authenticate will time out at 7365.196585s 2024-03-25T18:17:21.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.933+0000 7f302d039640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f30281430c0 0x7f302814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:34512/0 (socket says 172.21.15.122:34512) 2024-03-25T18:17:21.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.933+0000 7f302d039640 1 -- 172.21.15.122:0/926238334 learned_addr learned my addr 172.21.15.122:0/926238334 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:17:21.953 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.952+0000 7f301f7fe640 1 -- 172.21.15.122:0/926238334 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944266711 0 0) 0x7f3028142520 con 0x7f30280a98f0 2024-03-25T18:17:21.953 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.952+0000 7f301f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.953 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.952+0000 7f301f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.953 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.952+0000 7f301f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.953 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.952+0000 7f301f7fe640 10 monclient(hunting): my global_id is 4263 2024-03-25T18:17:21.953 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.952+0000 7f301f7fe640 1 -- 172.21.15.122:0/926238334 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3010003520 con 0x7f30280a98f0 2024-03-25T18:17:21.954 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.953+0000 7f301f7fe640 1 -- 172.21.15.122:0/926238334 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936561323 0 0) 0x7f30280b2a30 con 0x7f30281430c0 2024-03-25T18:17:21.955 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.953+0000 7f301f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.955 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.953+0000 7f301f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.955 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.953+0000 7f301f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.955 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.953+0000 7f301f7fe640 10 monclient(hunting): my global_id is 4247 2024-03-25T18:17:21.955 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.953+0000 7f301f7fe640 1 -- 172.21.15.122:0/926238334 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3028142520 con 0x7f30281430c0 2024-03-25T18:17:21.961 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.960+0000 7f301f7fe640 1 -- 172.21.15.122:0/926238334 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164661097 0 0) 0x7f30280b28a0 con 0x7f302813f710 2024-03-25T18:17:21.961 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.960+0000 7f301f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.961 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.960+0000 7f301f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.960+0000 7f301f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.960+0000 7f301f7fe640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:17:21.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.960+0000 7f301f7fe640 1 -- 172.21.15.122:0/926238334 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30280b2a30 con 0x7f302813f710 2024-03-25T18:17:21.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.969+0000 7f301f7fe640 1 -- 172.21.15.122:0/926238334 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3506890337 0 0) 0x7f3010003520 con 0x7f30280a98f0 2024-03-25T18:17:21.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.969+0000 7f301f7fe640 1 -- 172.21.15.122:0/926238334 >> v1:172.21.15.122:6790/0 conn(0x7f30281430c0 legacy=0x7f302814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.969+0000 7f301f7fe640 1 -- 172.21.15.122:0/926238334 >> v1:172.21.15.204:6789/0 conn(0x7f302813f710 legacy=0x7f30281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.969+0000 7f301f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:21.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.969+0000 7f301f7fe640 1 monclient: found mon.noname-a 2024-03-25T18:17:21.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.969+0000 7f301f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:21.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.969+0000 7f301f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:17:21.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.969+0000 7f301f7fe640 1 -- 172.21.15.122:0/926238334 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30280b44c0 con 0x7f30280a98f0 2024-03-25T18:17:21.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.969+0000 7f301f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:21.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.969+0000 7f301f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:21.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.969+0000 7f301f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:21.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.969+0000 7f301f7fe640 1 -- 172.21.15.122:0/926238334 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f3024001690 con 0x7f30280a98f0 2024-03-25T18:17:21.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.969+0000 7f302ea9c640 5 monclient: authenticate success, global_id 4263 2024-03-25T18:17:21.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.969+0000 7f301f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:21.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.969+0000 7f301f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:21.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.969+0000 7f301f7fe640 10 monclient: dump: 2024-03-25T18:17:21.971 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:21.971 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:21.971 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:21.971 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:21.971 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:21.972 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:21.972 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:21.972 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:21.972 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:21.972 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:21.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.969+0000 7f302ea9c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:21.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.983+0000 7f301f7fe640 1 -- 172.21.15.122:0/926238334 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f30240017f0 con 0x7f30280a98f0 2024-03-25T18:17:21.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.983+0000 7f301f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:21.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.983+0000 7f301f7fe640 1 -- 172.21.15.122:0/926238334 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f3024001c00 con 0x7f30280a98f0 2024-03-25T18:17:21.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.983+0000 7f301f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:21.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.983+0000 7f301f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:21.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.983+0000 7f301f7fe640 10 monclient: dump: 2024-03-25T18:17:21.984 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:21.984 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:21.984 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:21.985 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:21.985 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:21.985 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:21.985 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:21.985 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:21.985 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:21.985 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:21.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.983+0000 7f302ea9c640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:21.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.983+0000 7f302ea9c640 4 set_mon_vals no callback set 2024-03-25T18:17:21.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.984+0000 7f302ea9c640 10 monclient: shutdown 2024-03-25T18:17:21.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.984+0000 7f302ea9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:21.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.984+0000 7f302ea9c640 1 -- 172.21.15.122:0/926238334 >> v1:172.21.15.122:6789/0 conn(0x7f30280a98f0 legacy=0x7f30280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.984+0000 7f302ea9c640 1 -- 172.21.15.122:0/926238334 shutdown_connections 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.984+0000 7f302ea9c640 1 -- 172.21.15.122:0/926238334 >> 172.21.15.122:0/926238334 conn(0x7f30280b7ea0 msgr2=0x7f30280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.984+0000 7f302ea9c640 1 -- 172.21.15.122:0/926238334 shutdown_connections 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.984+0000 7f302ea9c640 1 -- 172.21.15.122:0/926238334 wait complete. 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.985+0000 7f302ea9c640 10 monclient: build_initial_monmap 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.985+0000 7f302ea9c640 10 monclient: monmap: 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.985+0000 7f302ea9c640 1 librados: starting msgr at 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.985+0000 7f302ea9c640 1 librados: starting objecter 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.985+0000 7f302ea9c640 1 Processor -- start 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.985+0000 7f302ea9c640 1 -- start start 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.985+0000 7f302ea9c640 1 librados: setting wanted keys 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.985+0000 7f302ea9c640 1 librados: calling monclient init 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.985+0000 7f302ea9c640 10 monclient: init 2024-03-25T18:17:21.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.986+0000 7f302ea9c640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:21.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.986+0000 7f302ea9c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:21.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.986+0000 7f302ea9c640 10 monclient(hunting): picked mon.noname-c con 0x7f30280a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:17:21.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.986+0000 7f302ea9c640 10 monclient(hunting): picked mon.noname-a con 0x7f302813f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:17:21.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.986+0000 7f302ea9c640 10 monclient(hunting): picked mon.noname-b con 0x7f30281430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:17:21.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.986+0000 7f302ea9c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3028132a30 con 0x7f302813f710 2024-03-25T18:17:21.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.986+0000 7f302ea9c640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3028138c80 con 0x7f30281430c0 2024-03-25T18:17:21.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.986+0000 7f302ea9c640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3028138f60 con 0x7f30280a98f0 2024-03-25T18:17:21.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.986+0000 7f302ea9c640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:21.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.986+0000 7f302ea9c640 10 monclient(hunting): authenticate will time out at 7365.252701s 2024-03-25T18:17:21.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.991+0000 7f302d039640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f30281430c0 0x7f3028132320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:53860/0 (socket says 172.21.15.122:53860) 2024-03-25T18:17:21.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.991+0000 7f302d039640 1 -- 172.21.15.122:0/2701166928 learned_addr learned my addr 172.21.15.122:0/2701166928 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:17:21.994 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.993+0000 7f301d7fa640 10 client.?.objecter ms_handle_connect 0x7f30281430c0 2024-03-25T18:17:21.994 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.993+0000 7f301d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:21.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.996+0000 7f301d7fa640 10 client.?.objecter ms_handle_connect 0x7f302813f710 2024-03-25T18:17:21.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.996+0000 7f301d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:21.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.996+0000 7f301d7fa640 10 client.?.objecter ms_handle_connect 0x7f30280a98f0 2024-03-25T18:17:21.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:21.996+0000 7f301d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:22.002 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.001+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692802113 0 0) 0x7f3028138c80 con 0x7f30281430c0 2024-03-25T18:17:22.002 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.001+0000 7f301d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:22.002 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.001+0000 7f301d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:22.002 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.001+0000 7f301d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:22.002 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.001+0000 7f301d7fa640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:17:22.002 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.001+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3014004eb0 con 0x7f30281430c0 2024-03-25T18:17:22.010 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.009+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051658396 0 0) 0x7f3028138f60 con 0x7f30280a98f0 2024-03-25T18:17:22.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.009+0000 7f301d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:22.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.009+0000 7f301d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:22.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.009+0000 7f301d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:22.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.009+0000 7f301d7fa640 10 monclient(hunting): my global_id is 4250 2024-03-25T18:17:22.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.010+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3028138c80 con 0x7f30280a98f0 2024-03-25T18:17:22.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.012+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619033971 0 0) 0x7f3028132a30 con 0x7f302813f710 2024-03-25T18:17:22.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.012+0000 7f301d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:22.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.012+0000 7f301d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:22.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.012+0000 7f301d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:22.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.012+0000 7f301d7fa640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:17:22.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.012+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3028138f60 con 0x7f302813f710 2024-03-25T18:17:22.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.021+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 647754929 0 0) 0x7f3028138c80 con 0x7f30280a98f0 2024-03-25T18:17:22.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.021+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3028132a30 con 0x7f30280a98f0 2024-03-25T18:17:22.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.021+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f3024001b00 con 0x7f30280a98f0 2024-03-25T18:17:22.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.021+0000 7f301d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:22.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.021+0000 7f301d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:22.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.021+0000 7f301d7fa640 10 monclient(hunting): dump: 2024-03-25T18:17:22.022 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:22.022 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:22.023 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:22.023 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:22.023 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:22.023 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:22.023 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:22.023 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:22.023 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:22.023 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:22.023 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.021+0000 7f301d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:22.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.022+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3012922722 0 0) 0x7f3014004eb0 con 0x7f30281430c0 2024-03-25T18:17:22.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.022+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3028138c80 con 0x7f30281430c0 2024-03-25T18:17:22.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.022+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f3020001c70 con 0x7f30281430c0 2024-03-25T18:17:22.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.022+0000 7f301d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:22.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.022+0000 7f301d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:22.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.022+0000 7f301d7fa640 10 monclient(hunting): dump: 2024-03-25T18:17:22.024 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:22.024 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:22.024 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:22.024 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:22.024 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:22.024 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:22.024 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:22.025 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:22.025 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:22.025 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:22.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 998920741 0 0) 0x7f3028138c80 con 0x7f30281430c0 2024-03-25T18:17:22.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 >> v1:172.21.15.122:6790/0 conn(0x7f30280a98f0 legacy=0x7f3028137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:22.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 >> v1:172.21.15.122:6789/0 conn(0x7f302813f710 legacy=0x7f3028131c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:22.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f301d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:17:22.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f301d7fa640 1 monclient: found mon.b 2024-03-25T18:17:22.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f301d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:22.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f301d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:22.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3028139240 con 0x7f30281430c0 2024-03-25T18:17:22.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f301d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:17:22.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f301d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:17:22.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f301d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:22.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f302ea9c640 5 monclient: authenticate success, global_id 4264 2024-03-25T18:17:22.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f302ea9c640 10 monclient: _renew_subs 2024-03-25T18:17:22.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f302ea9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:22.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f302ea9c640 1 -- 172.21.15.122:0/2701166928 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3028136e50 con 0x7f30281430c0 2024-03-25T18:17:22.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f302ea9c640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:22.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f302ea9c640 10 monclient: _renew_subs 2024-03-25T18:17:22.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f302ea9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:22.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f302ea9c640 1 -- 172.21.15.122:0/2701166928 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30281389f0 con 0x7f30281430c0 2024-03-25T18:17:22.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.034+0000 7f302ea9c640 1 librados: init done 2024-03-25T18:17:22.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.036+0000 7f302ea9c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:22.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.036+0000 7f302ea9c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:22.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.036+0000 7f302ea9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:22.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.036+0000 7f302ea9c640 1 -- 172.21.15.122:0/2701166928 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30280627b0 con 0x7f30281430c0 2024-03-25T18:17:22.316 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.315+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3020004b60 con 0x7f30281430c0 2024-03-25T18:17:22.317 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.315+0000 7f301d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:22.317 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.315+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f3020004e70 con 0x7f30281430c0 2024-03-25T18:17:22.317 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.315+0000 7f301d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:22.317 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.315+0000 7f301d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:22.317 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.315+0000 7f301d7fa640 10 monclient: dump: 2024-03-25T18:17:22.317 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:22.317 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:22.317 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:22.317 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:22.317 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:22.318 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:22.318 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:22.318 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:22.318 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:22.318 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:22.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.315+0000 7f301effd640 4 set_mon_vals no callback set 2024-03-25T18:17:22.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.316+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f302001ae40 con 0x7f30281430c0 2024-03-25T18:17:22.319 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.317+0000 7f302c838640 10 monclient: get_auth_request con 0x7f301406dfb0 auth_method 0 2024-03-25T18:17:22.319 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.317+0000 7f301d7fa640 10 client.4264.objecter ms_handle_connect 0x7f301406dfb0 2024-03-25T18:17:22.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.320+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f3020054b80 con 0x7f30281430c0 2024-03-25T18:17:22.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.321+0000 7f301d7fa640 10 client.4264.objecter ms_dispatch 0x7f3028000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:22.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.321+0000 7f301d7fa640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:22.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.321+0000 7f301d7fa640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:22.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.321+0000 7f301d7fa640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-25T18:17:22.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.321+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f30200504c0 con 0x7f30281430c0 2024-03-25T18:17:22.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.321+0000 7f301d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:22.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.321+0000 7f301d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:22.577 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.575+0000 7f302ea9c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:22.578 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.575+0000 7f302ea9c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:22.578 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.575+0000 7f302ea9c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:22.578 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.575+0000 7f302ea9c640 1 -- 172.21.15.122:0/2701166928 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f302813dd60 con 0x7f30281430c0 2024-03-25T18:17:22.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.605+0000 7f301d7fa640 1 -- 172.21.15.122:0/2701166928 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f30200525c0 con 0x7f30281430c0 2024-03-25T18:17:22.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.605+0000 7f301d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:22.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.605+0000 7f301d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:22.606 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:17:22.606 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:22.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.607+0000 7f302ea9c640 10 librados: watch_flush enter 2024-03-25T18:17:22.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.607+0000 7f302ea9c640 10 librados: watch_flush exit 2024-03-25T18:17:22.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.607+0000 7f302ea9c640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-25T18:17:22.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.607+0000 7f302ea9c640 10 client.4264.objecter successfully canceled tick 2024-03-25T18:17:22.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.607+0000 7f302ea9c640 1 -- 172.21.15.122:0/2701166928 >> v1:172.21.15.204:6800/676228005 conn(0x7f301406dfb0 legacy=0x7f3014070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:22.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.607+0000 7f302ea9c640 10 monclient: shutdown 2024-03-25T18:17:22.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.607+0000 7f302ea9c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:22.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.607+0000 7f302ea9c640 1 -- 172.21.15.122:0/2701166928 >> v1:172.21.15.204:6789/0 conn(0x7f30281430c0 legacy=0x7f3028132320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:22.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.607+0000 7f302ea9c640 1 -- 172.21.15.122:0/2701166928 shutdown_connections 2024-03-25T18:17:22.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.607+0000 7f302ea9c640 1 -- 172.21.15.122:0/2701166928 >> 172.21.15.122:0/2701166928 conn(0x7f30280b7ea0 msgr2=0x7f3028074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:22.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.607+0000 7f302ea9c640 1 -- 172.21.15.122:0/2701166928 shutdown_connections 2024-03-25T18:17:22.610 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.607+0000 7f302ea9c640 1 -- 172.21.15.122:0/2701166928 wait complete. 2024-03-25T18:17:22.610 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:22.607+0000 7f302ea9c640 1 librados: shutdown 2024-03-25T18:17:22.620 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:25.329 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:01:19.071 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:25.397 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:01:19.140 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:25.408 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:01:19.150 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:25.408 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:01:19.154 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:25.476 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:01:19.222 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:25.480 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:01:19.222 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:25.641 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:01:19.383 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:25.700 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:01:19.445 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:25.726 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:01:19.469 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:29.426 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:29.543 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.542+0000 7f143eb6c640 10 monclient: get_monmap_and_config 2024-03-25T18:17:29.543 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.542+0000 7f143eb6c640 10 monclient: build_initial_monmap 2024-03-25T18:17:29.544 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.542+0000 7f143eb6c640 10 monclient: monmap: 2024-03-25T18:17:29.544 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:17:29.544 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:29.544 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:17:29.543804+0000 2024-03-25T18:17:29.544 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:17:29.543804+0000 2024-03-25T18:17:29.544 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:29.544 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:29.544 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:17:29.544 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:17:29.544 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:17:29.545 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:29.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.543+0000 7f143eb6c640 1 Processor -- start 2024-03-25T18:17:29.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.543+0000 7f143eb6c640 1 -- start start 2024-03-25T18:17:29.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.543+0000 7f143eb6c640 10 monclient: init 2024-03-25T18:17:29.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.544+0000 7f143eb6c640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:29.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.544+0000 7f143eb6c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:29.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.544+0000 7f143eb6c640 10 monclient(hunting): picked mon.noname-b con 0x7f14380705b0 addr v1:172.21.15.122:6790/0 2024-03-25T18:17:29.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.544+0000 7f143eb6c640 10 monclient(hunting): picked mon.noname-a con 0x7f14381402d0 addr v1:172.21.15.122:6789/0 2024-03-25T18:17:29.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.544+0000 7f143eb6c640 10 monclient(hunting): picked mon.noname-c con 0x7f1438149ae0 addr v1:172.21.15.204:6789/0 2024-03-25T18:17:29.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.544+0000 7f143eb6c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14380718c0 con 0x7f14381402d0 2024-03-25T18:17:29.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.544+0000 7f143eb6c640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14380714c0 con 0x7f1438149ae0 2024-03-25T18:17:29.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.544+0000 7f143eb6c640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1438072b50 con 0x7f14380705b0 2024-03-25T18:17:29.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.544+0000 7f143eb6c640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:29.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.544+0000 7f143eb6c640 10 monclient(hunting): authenticate will time out at 7372.810568s 2024-03-25T18:17:29.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.567+0000 7f143c908640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f14380705b0 0x7f143813d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:36158/0 (socket says 172.21.15.122:36158) 2024-03-25T18:17:29.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.567+0000 7f143c908640 1 -- 172.21.15.122:0/4104521165 learned_addr learned my addr 172.21.15.122:0/4104521165 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:17:29.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.584+0000 7f14377fe640 1 -- 172.21.15.122:0/4104521165 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529608548 0 0) 0x7f14380714c0 con 0x7f1438149ae0 2024-03-25T18:17:29.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.584+0000 7f14377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:29.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.584+0000 7f14377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:29.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.584+0000 7f14377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:29.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.584+0000 7f14377fe640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:17:29.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.584+0000 7f14377fe640 1 -- 172.21.15.122:0/4104521165 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1428003520 con 0x7f1438149ae0 2024-03-25T18:17:29.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.584+0000 7f14377fe640 1 -- 172.21.15.122:0/4104521165 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 128067614 0 0) 0x7f14380718c0 con 0x7f14381402d0 2024-03-25T18:17:29.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.584+0000 7f14377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:29.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.584+0000 7f14377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:29.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.584+0000 7f14377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:29.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.584+0000 7f14377fe640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:17:29.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.584+0000 7f14377fe640 1 -- 172.21.15.122:0/4104521165 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14380714c0 con 0x7f14381402d0 2024-03-25T18:17:29.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.586+0000 7f14377fe640 1 -- 172.21.15.122:0/4104521165 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197942970 0 0) 0x7f1438072b50 con 0x7f14380705b0 2024-03-25T18:17:29.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.586+0000 7f14377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:29.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.586+0000 7f14377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:29.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.586+0000 7f14377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:29.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.586+0000 7f14377fe640 10 monclient(hunting): my global_id is 4253 2024-03-25T18:17:29.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.586+0000 7f14377fe640 1 -- 172.21.15.122:0/4104521165 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14380718c0 con 0x7f14380705b0 2024-03-25T18:17:29.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.592+0000 7f14377fe640 1 -- 172.21.15.122:0/4104521165 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4212419315 0 0) 0x7f1428003520 con 0x7f1438149ae0 2024-03-25T18:17:29.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.592+0000 7f14377fe640 1 -- 172.21.15.122:0/4104521165 >> v1:172.21.15.122:6790/0 conn(0x7f14380705b0 legacy=0x7f143813d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.592+0000 7f14377fe640 1 -- 172.21.15.122:0/4104521165 >> v1:172.21.15.122:6789/0 conn(0x7f14381402d0 legacy=0x7f1438140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.593 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.592+0000 7f14377fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:29.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.592+0000 7f14377fe640 1 monclient: found mon.noname-c 2024-03-25T18:17:29.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.592+0000 7f14377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:29.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.592+0000 7f14377fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:17:29.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.592+0000 7f14377fe640 1 -- 172.21.15.122:0/4104521165 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143813dd10 con 0x7f1438149ae0 2024-03-25T18:17:29.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.592+0000 7f14377fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:29.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.592+0000 7f14377fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:29.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.592+0000 7f14377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:29.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.592+0000 7f143eb6c640 5 monclient: authenticate success, global_id 4267 2024-03-25T18:17:29.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.592+0000 7f143eb6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:29.596 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.595+0000 7f14377fe640 1 -- 172.21.15.122:0/4104521165 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f14300019d0 con 0x7f1438149ae0 2024-03-25T18:17:29.596 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.595+0000 7f14377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:29.596 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.595+0000 7f14377fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:29.596 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.595+0000 7f14377fe640 10 monclient: dump: 2024-03-25T18:17:29.596 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:29.596 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:29.596 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:29.596 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:29.596 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:29.596 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:29.596 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:29.597 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:29.597 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:29.597 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:29.597 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.595+0000 7f143eb6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:29.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.607+0000 7f14377fe640 1 -- 172.21.15.122:0/4104521165 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1430001c20 con 0x7f1438149ae0 2024-03-25T18:17:29.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.607+0000 7f14377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:29.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.607+0000 7f14377fe640 1 -- 172.21.15.122:0/4104521165 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f1430001fb0 con 0x7f1438149ae0 2024-03-25T18:17:29.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.607+0000 7f14377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:29.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.607+0000 7f14377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:29.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.607+0000 7f14377fe640 10 monclient: dump: 2024-03-25T18:17:29.609 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:29.609 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:29.609 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:29.609 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:29.609 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:29.609 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:29.609 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:29.609 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:29.609 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:29.609 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:29.610 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.607+0000 7f143eb6c640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:29.610 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.607+0000 7f143eb6c640 4 set_mon_vals no callback set 2024-03-25T18:17:29.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.608+0000 7f143eb6c640 10 monclient: shutdown 2024-03-25T18:17:29.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.608+0000 7f143eb6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:29.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.608+0000 7f143eb6c640 1 -- 172.21.15.122:0/4104521165 >> v1:172.21.15.204:6789/0 conn(0x7f1438149ae0 legacy=0x7f143814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.608+0000 7f143eb6c640 1 -- 172.21.15.122:0/4104521165 shutdown_connections 2024-03-25T18:17:29.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.608+0000 7f143eb6c640 1 -- 172.21.15.122:0/4104521165 >> 172.21.15.122:0/4104521165 conn(0x7f14380b5d10 msgr2=0x7f14380b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:29.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.608+0000 7f143eb6c640 1 -- 172.21.15.122:0/4104521165 shutdown_connections 2024-03-25T18:17:29.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.608+0000 7f143eb6c640 1 -- 172.21.15.122:0/4104521165 wait complete. 2024-03-25T18:17:29.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.609+0000 7f143eb6c640 10 monclient: build_initial_monmap 2024-03-25T18:17:29.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.609+0000 7f143eb6c640 10 monclient: monmap: 2024-03-25T18:17:29.612 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:17:29.612 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:29.612 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:17:29.612 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:17:29.612 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:29.612 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:17:29.612 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:17:29.612 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:17:29.612 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:17:29.612 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:29.612 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.610+0000 7f143eb6c640 1 librados: starting msgr at 2024-03-25T18:17:29.612 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.610+0000 7f143eb6c640 1 librados: starting objecter 2024-03-25T18:17:29.612 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.610+0000 7f143eb6c640 1 Processor -- start 2024-03-25T18:17:29.612 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.610+0000 7f143eb6c640 1 -- start start 2024-03-25T18:17:29.612 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.610+0000 7f143eb6c640 1 librados: setting wanted keys 2024-03-25T18:17:29.613 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.610+0000 7f143eb6c640 1 librados: calling monclient init 2024-03-25T18:17:29.613 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.610+0000 7f143eb6c640 10 monclient: init 2024-03-25T18:17:29.613 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.611+0000 7f143eb6c640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:29.613 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.611+0000 7f143eb6c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:29.613 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.611+0000 7f143eb6c640 10 monclient(hunting): picked mon.noname-c con 0x7f14380705b0 addr v1:172.21.15.122:6790/0 2024-03-25T18:17:29.613 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.611+0000 7f143eb6c640 10 monclient(hunting): picked mon.noname-b con 0x7f14381402d0 addr v1:172.21.15.204:6789/0 2024-03-25T18:17:29.613 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.611+0000 7f143eb6c640 10 monclient(hunting): picked mon.noname-a con 0x7f1438149ae0 addr v1:172.21.15.122:6789/0 2024-03-25T18:17:29.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.611+0000 7f143eb6c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1438134a70 con 0x7f1438149ae0 2024-03-25T18:17:29.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.611+0000 7f143eb6c640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1438144240 con 0x7f14381402d0 2024-03-25T18:17:29.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.611+0000 7f143eb6c640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1438141550 con 0x7f14380705b0 2024-03-25T18:17:29.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.611+0000 7f143eb6c640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:29.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.611+0000 7f143eb6c640 10 monclient(hunting): authenticate will time out at 7372.877317s 2024-03-25T18:17:29.616 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.615+0000 7f143c908640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f14380705b0 0x7f1438144b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:36194/0 (socket says 172.21.15.122:36194) 2024-03-25T18:17:29.616 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.615+0000 7f143c908640 1 -- 172.21.15.122:0/529296704 learned_addr learned my addr 172.21.15.122:0/529296704 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:17:29.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.616+0000 7f14357fa640 10 client.?.objecter ms_handle_connect 0x7f14380705b0 2024-03-25T18:17:29.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.616+0000 7f14357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:29.625 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.624+0000 7f14357fa640 10 client.?.objecter ms_handle_connect 0x7f14381402d0 2024-03-25T18:17:29.625 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.624+0000 7f14357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:29.626 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.625+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944909723 0 0) 0x7f1438141550 con 0x7f14380705b0 2024-03-25T18:17:29.626 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.625+0000 7f14357fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:29.626 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.625+0000 7f14357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:29.626 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.625+0000 7f14357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:29.626 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.625+0000 7f14357fa640 10 monclient(hunting): my global_id is 4256 2024-03-25T18:17:29.626 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.625+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f141c004eb0 con 0x7f14380705b0 2024-03-25T18:17:29.631 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.630+0000 7f14357fa640 10 client.?.objecter ms_handle_connect 0x7f1438149ae0 2024-03-25T18:17:29.631 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.630+0000 7f14357fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:29.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.639+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590713871 0 0) 0x7f1438144240 con 0x7f14381402d0 2024-03-25T18:17:29.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.639+0000 7f14357fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:29.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.639+0000 7f14357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:29.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.639+0000 7f14357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:29.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.639+0000 7f14357fa640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:17:29.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.639+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1438141550 con 0x7f14381402d0 2024-03-25T18:17:29.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.648+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3842112316 0 0) 0x7f141c004eb0 con 0x7f14380705b0 2024-03-25T18:17:29.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.648+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1438144240 con 0x7f14380705b0 2024-03-25T18:17:29.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.648+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f142c001910 con 0x7f14380705b0 2024-03-25T18:17:29.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.648+0000 7f14357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:29.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.648+0000 7f14357fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:29.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.648+0000 7f14357fa640 10 monclient(hunting): dump: 2024-03-25T18:17:29.650 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:29.650 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:29.650 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:29.650 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:29.650 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:29.651 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:29.651 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:29.651 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:29.651 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:29.651 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:29.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.648+0000 7f14357fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:29.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.648+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562065878 0 0) 0x7f1438134a70 con 0x7f1438149ae0 2024-03-25T18:17:29.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.648+0000 7f14357fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:29.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.648+0000 7f14357fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:29.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.648+0000 7f14357fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:29.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.649+0000 7f14357fa640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:17:29.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.649+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f141c004eb0 con 0x7f1438149ae0 2024-03-25T18:17:29.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.652+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1011515375 0 0) 0x7f1438141550 con 0x7f14381402d0 2024-03-25T18:17:29.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.652+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1438134a70 con 0x7f14381402d0 2024-03-25T18:17:29.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.652+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f14240018b0 con 0x7f14381402d0 2024-03-25T18:17:29.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.652+0000 7f14357fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:29.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.652+0000 7f14357fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:29.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.652+0000 7f14357fa640 10 monclient(hunting): dump: 2024-03-25T18:17:29.654 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:29.654 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:29.654 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:29.654 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:29.654 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:29.654 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:29.656 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2944082322 0 0) 0x7f1438144240 con 0x7f14380705b0 2024-03-25T18:17:29.656 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 >> v1:172.21.15.204:6789/0 conn(0x7f14381402d0 legacy=0x7f1438133c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.656 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 >> v1:172.21.15.122:6789/0 conn(0x7f1438149ae0 legacy=0x7f1438134360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.656 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f14357fa640 10 monclient: _finish_hunting 0 2024-03-25T18:17:29.656 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f14357fa640 1 monclient: found mon.c 2024-03-25T18:17:29.656 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f14357fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:29.656 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f14357fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:17:29.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1438145230 con 0x7f14380705b0 2024-03-25T18:17:29.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f14357fa640 10 monclient: _finish_auth 0 2024-03-25T18:17:29.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f14357fa640 10 monclient: _check_auth_tickets 2024-03-25T18:17:29.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f14357fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:29.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f143eb6c640 5 monclient: authenticate success, global_id 4256 2024-03-25T18:17:29.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f143eb6c640 10 monclient: _renew_subs 2024-03-25T18:17:29.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f143eb6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:17:29.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f143eb6c640 1 -- 172.21.15.122:0/529296704 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14381426c0 con 0x7f14380705b0 2024-03-25T18:17:29.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f143eb6c640 10 client.4256.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:29.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f143eb6c640 10 monclient: _renew_subs 2024-03-25T18:17:29.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f143eb6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:17:29.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f143eb6c640 1 -- 172.21.15.122:0/529296704 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f143813fa00 con 0x7f14380705b0 2024-03-25T18:17:29.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.655+0000 7f143eb6c640 1 librados: init done 2024-03-25T18:17:29.658 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.656+0000 7f143eb6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:29.658 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.656+0000 7f143eb6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:29.658 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.656+0000 7f143eb6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:17:29.658 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.656+0000 7f143eb6c640 1 -- 172.21.15.122:0/529296704 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14380627b0 con 0x7f14380705b0 2024-03-25T18:17:29.949 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.947+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f142c0047e0 con 0x7f14380705b0 2024-03-25T18:17:29.949 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.947+0000 7f14357fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:29.949 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.947+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f142c004b70 con 0x7f14380705b0 2024-03-25T18:17:29.949 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.947+0000 7f14357fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:29.950 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.947+0000 7f14357fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:29.950 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.947+0000 7f14357fa640 10 monclient: dump: 2024-03-25T18:17:29.950 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:29.950 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:29.950 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:29.950 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:29.950 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:29.950 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:29.950 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:29.950 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:29.950 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:29.950 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:29.950 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.947+0000 7f1436ffd640 4 set_mon_vals no callback set 2024-03-25T18:17:29.951 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.948+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f142c004d90 con 0x7f14380705b0 2024-03-25T18:17:29.951 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.949+0000 7f1437fff640 10 monclient: get_auth_request con 0x7f141c06df10 auth_method 0 2024-03-25T18:17:29.951 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.949+0000 7f14357fa640 10 client.4256.objecter ms_handle_connect 0x7f141c06df10 2024-03-25T18:17:29.955 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.954+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f142c054820 con 0x7f14380705b0 2024-03-25T18:17:29.956 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.954+0000 7f14357fa640 10 client.4256.objecter ms_dispatch 0x7f1438000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:29.956 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.954+0000 7f14357fa640 3 client.4256.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:29.956 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.954+0000 7f14357fa640 3 client.4256.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:29.956 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.954+0000 7f14357fa640 20 client.4256.objecter dump_active .. 0 homeless 2024-03-25T18:17:29.956 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.954+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f142c050330 con 0x7f14380705b0 2024-03-25T18:17:29.956 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.954+0000 7f14357fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:29.956 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:29.954+0000 7f14357fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:30.203 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.202+0000 7f143eb6c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:30.204 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.202+0000 7f143eb6c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:30.204 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.202+0000 7f143eb6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:17:30.204 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.202+0000 7f143eb6c640 1 -- 172.21.15.122:0/529296704 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1438142bc0 con 0x7f14380705b0 2024-03-25T18:17:30.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.233+0000 7f14357fa640 1 -- 172.21.15.122:0/529296704 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f142c052260 con 0x7f14380705b0 2024-03-25T18:17:30.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.233+0000 7f14357fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:30.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.233+0000 7f14357fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:30.234 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:17:30.234 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:30.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.235+0000 7f143eb6c640 10 librados: watch_flush enter 2024-03-25T18:17:30.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.235+0000 7f143eb6c640 10 librados: watch_flush exit 2024-03-25T18:17:30.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.235+0000 7f143eb6c640 20 client.4256.objecter shutdown clearing up homeless session... 2024-03-25T18:17:30.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.235+0000 7f143eb6c640 10 client.4256.objecter successfully canceled tick 2024-03-25T18:17:30.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.235+0000 7f143eb6c640 1 -- 172.21.15.122:0/529296704 >> v1:172.21.15.204:6800/676228005 conn(0x7f141c06df10 legacy=0x7f141c0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:30.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.235+0000 7f143eb6c640 10 monclient: shutdown 2024-03-25T18:17:30.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.235+0000 7f143eb6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:30.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.235+0000 7f143eb6c640 1 -- 172.21.15.122:0/529296704 >> v1:172.21.15.122:6790/0 conn(0x7f14380705b0 legacy=0x7f1438144b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:30.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.235+0000 7f143eb6c640 1 -- 172.21.15.122:0/529296704 shutdown_connections 2024-03-25T18:17:30.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.235+0000 7f143eb6c640 1 -- 172.21.15.122:0/529296704 >> 172.21.15.122:0/529296704 conn(0x7f14380b5d10 msgr2=0x7f1438140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:30.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.235+0000 7f143eb6c640 1 -- 172.21.15.122:0/529296704 shutdown_connections 2024-03-25T18:17:30.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.235+0000 7f143eb6c640 1 -- 172.21.15.122:0/529296704 wait complete. 2024-03-25T18:17:30.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:30.236+0000 7f143eb6c640 1 librados: shutdown 2024-03-25T18:17:30.248 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:37.054 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:37.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.162+0000 7efc66fef640 10 monclient: get_monmap_and_config 2024-03-25T18:17:37.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.162+0000 7efc66fef640 10 monclient: build_initial_monmap 2024-03-25T18:17:37.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.162+0000 7efc66fef640 10 monclient: monmap: 2024-03-25T18:17:37.164 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:17:37.164 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:37.164 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:17:37.163838+0000 2024-03-25T18:17:37.164 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:17:37.163838+0000 2024-03-25T18:17:37.164 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:37.164 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:37.164 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:17:37.164 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:17:37.164 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:17:37.164 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:37.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.163+0000 7efc66fef640 1 Processor -- start 2024-03-25T18:17:37.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.164+0000 7efc66fef640 1 -- start start 2024-03-25T18:17:37.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.164+0000 7efc66fef640 10 monclient: init 2024-03-25T18:17:37.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.165+0000 7efc66fef640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:37.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.165+0000 7efc66fef640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:37.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.165+0000 7efc66fef640 10 monclient(hunting): picked mon.noname-c con 0x7efc600a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:17:37.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.165+0000 7efc66fef640 10 monclient(hunting): picked mon.noname-b con 0x7efc6013f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:17:37.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.165+0000 7efc66fef640 10 monclient(hunting): picked mon.noname-a con 0x7efc601430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:17:37.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.165+0000 7efc66fef640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc60142520 con 0x7efc601430c0 2024-03-25T18:17:37.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.165+0000 7efc66fef640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc600b28a0 con 0x7efc600a98f0 2024-03-25T18:17:37.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.165+0000 7efc66fef640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc600b2a30 con 0x7efc6013f710 2024-03-25T18:17:37.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.165+0000 7efc66fef640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:37.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.165+0000 7efc66fef640 10 monclient(hunting): authenticate will time out at 7380.431700s 2024-03-25T18:17:37.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.178+0000 7efc6558c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7efc601430c0 0x7efc6014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45616/0 (socket says 172.21.15.122:45616) 2024-03-25T18:17:37.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.178+0000 7efc6558c640 1 -- 172.21.15.122:0/2221731838 learned_addr learned my addr 172.21.15.122:0/2221731838 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:17:37.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.198+0000 7efc5f7fe640 1 -- 172.21.15.122:0/2221731838 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1021584785 0 0) 0x7efc60142520 con 0x7efc601430c0 2024-03-25T18:17:37.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.199+0000 7efc5f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:37.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.199+0000 7efc5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:37.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.199+0000 7efc5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:37.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.199+0000 7efc5f7fe640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:17:37.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.199+0000 7efc5f7fe640 1 -- 172.21.15.122:0/2221731838 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc50003520 con 0x7efc601430c0 2024-03-25T18:17:37.202 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.201+0000 7efc5f7fe640 1 -- 172.21.15.122:0/2221731838 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 249697892 0 0) 0x7efc600b28a0 con 0x7efc600a98f0 2024-03-25T18:17:37.202 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.201+0000 7efc5f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:37.202 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.201+0000 7efc5f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:37.202 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.201+0000 7efc5f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:37.202 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.201+0000 7efc5f7fe640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:17:37.202 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.201+0000 7efc5f7fe640 1 -- 172.21.15.122:0/2221731838 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc60142520 con 0x7efc600a98f0 2024-03-25T18:17:37.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.213+0000 7efc5f7fe640 1 -- 172.21.15.122:0/2221731838 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 476378752 0 0) 0x7efc60142520 con 0x7efc600a98f0 2024-03-25T18:17:37.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.213+0000 7efc5f7fe640 1 -- 172.21.15.122:0/2221731838 >> v1:172.21.15.122:6790/0 conn(0x7efc6013f710 legacy=0x7efc601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.213+0000 7efc5f7fe640 1 -- 172.21.15.122:0/2221731838 >> v1:172.21.15.122:6789/0 conn(0x7efc601430c0 legacy=0x7efc6014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.213+0000 7efc5f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:37.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.213+0000 7efc5f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:17:37.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.213+0000 7efc5f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:37.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.213+0000 7efc5f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:17:37.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.213+0000 7efc5f7fe640 1 -- 172.21.15.122:0/2221731838 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc600b44c0 con 0x7efc600a98f0 2024-03-25T18:17:37.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.213+0000 7efc5f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:37.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.213+0000 7efc5f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:37.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.213+0000 7efc5f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:37.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.213+0000 7efc5f7fe640 1 -- 172.21.15.122:0/2221731838 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7efc54001670 con 0x7efc600a98f0 2024-03-25T18:17:37.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.213+0000 7efc5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:37.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.213+0000 7efc5f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:37.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.213+0000 7efc5f7fe640 10 monclient: dump: 2024-03-25T18:17:37.216 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:37.216 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:37.216 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:37.216 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:37.216 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:37.216 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:37.216 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:37.216 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:37.216 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:37.216 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:37.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.213+0000 7efc66fef640 5 monclient: authenticate success, global_id 4273 2024-03-25T18:17:37.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.213+0000 7efc66fef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:37.222 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.220+0000 7efc5f7fe640 1 -- 172.21.15.122:0/2221731838 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc540017d0 con 0x7efc600a98f0 2024-03-25T18:17:37.222 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.220+0000 7efc5f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:37.222 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.220+0000 7efc5f7fe640 1 -- 172.21.15.122:0/2221731838 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7efc54001ba0 con 0x7efc600a98f0 2024-03-25T18:17:37.222 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.220+0000 7efc66fef640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:37.222 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.220+0000 7efc66fef640 4 set_mon_vals no callback set 2024-03-25T18:17:37.222 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.220+0000 7efc5f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:37.222 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.220+0000 7efc5f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:37.222 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.220+0000 7efc5f7fe640 10 monclient: dump: 2024-03-25T18:17:37.222 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:37.222 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:37.222 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:37.223 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:37.223 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:37.223 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:37.223 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:37.223 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:37.223 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:37.223 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:37.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.221+0000 7efc66fef640 10 monclient: shutdown 2024-03-25T18:17:37.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.221+0000 7efc66fef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:37.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.221+0000 7efc66fef640 1 -- 172.21.15.122:0/2221731838 >> v1:172.21.15.204:6789/0 conn(0x7efc600a98f0 legacy=0x7efc600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.221+0000 7efc66fef640 1 -- 172.21.15.122:0/2221731838 shutdown_connections 2024-03-25T18:17:37.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.221+0000 7efc66fef640 1 -- 172.21.15.122:0/2221731838 >> 172.21.15.122:0/2221731838 conn(0x7efc600b7ea0 msgr2=0x7efc600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:37.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.221+0000 7efc66fef640 1 -- 172.21.15.122:0/2221731838 shutdown_connections 2024-03-25T18:17:37.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.221+0000 7efc66fef640 1 -- 172.21.15.122:0/2221731838 wait complete. 2024-03-25T18:17:37.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.222+0000 7efc66fef640 10 monclient: build_initial_monmap 2024-03-25T18:17:37.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.222+0000 7efc66fef640 10 monclient: monmap: 2024-03-25T18:17:37.225 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:17:37.225 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:37.226 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:17:37.226 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:17:37.226 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:37.226 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:17:37.226 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:17:37.226 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:17:37.226 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:17:37.226 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:37.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.222+0000 7efc66fef640 1 librados: starting msgr at 2024-03-25T18:17:37.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.222+0000 7efc66fef640 1 librados: starting objecter 2024-03-25T18:17:37.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.222+0000 7efc66fef640 1 Processor -- start 2024-03-25T18:17:37.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.222+0000 7efc66fef640 1 -- start start 2024-03-25T18:17:37.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.222+0000 7efc66fef640 1 librados: setting wanted keys 2024-03-25T18:17:37.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.222+0000 7efc66fef640 1 librados: calling monclient init 2024-03-25T18:17:37.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.222+0000 7efc66fef640 10 monclient: init 2024-03-25T18:17:37.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.223+0000 7efc66fef640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:37.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.223+0000 7efc66fef640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:37.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.223+0000 7efc66fef640 10 monclient(hunting): picked mon.noname-c con 0x7efc600a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:17:37.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.223+0000 7efc66fef640 10 monclient(hunting): picked mon.noname-b con 0x7efc6013f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:17:37.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.223+0000 7efc66fef640 10 monclient(hunting): picked mon.noname-a con 0x7efc601430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:17:37.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.223+0000 7efc66fef640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc60138940 con 0x7efc601430c0 2024-03-25T18:17:37.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.223+0000 7efc66fef640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc601d51f0 con 0x7efc6013f710 2024-03-25T18:17:37.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.223+0000 7efc66fef640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc601379e0 con 0x7efc600a98f0 2024-03-25T18:17:37.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.223+0000 7efc66fef640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:37.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.223+0000 7efc66fef640 10 monclient(hunting): authenticate will time out at 7380.489853s 2024-03-25T18:17:37.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.233+0000 7efc6558c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7efc601430c0 0x7efc601db1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:45684/0 (socket says 172.21.15.122:45684) 2024-03-25T18:17:37.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.233+0000 7efc6558c640 1 -- 172.21.15.122:0/4185174047 learned_addr learned my addr 172.21.15.122:0/4185174047 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:17:37.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.234+0000 7efc5d7fa640 10 client.?.objecter ms_handle_connect 0x7efc601430c0 2024-03-25T18:17:37.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.234+0000 7efc5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:37.244 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.243+0000 7efc5d7fa640 10 client.?.objecter ms_handle_connect 0x7efc6013f710 2024-03-25T18:17:37.244 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.243+0000 7efc5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:37.247 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.246+0000 7efc5d7fa640 10 client.?.objecter ms_handle_connect 0x7efc600a98f0 2024-03-25T18:17:37.247 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.246+0000 7efc5d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:37.249 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.248+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4197740734 0 0) 0x7efc601d51f0 con 0x7efc6013f710 2024-03-25T18:17:37.249 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.248+0000 7efc5d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:37.249 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.248+0000 7efc5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:37.249 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.248+0000 7efc5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:37.249 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.248+0000 7efc5d7fa640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:17:37.250 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.248+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc44004eb0 con 0x7efc6013f710 2024-03-25T18:17:37.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.252+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995576148 0 0) 0x7efc50003520 con 0x7efc601430c0 2024-03-25T18:17:37.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.252+0000 7efc5d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:37.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.252+0000 7efc5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:37.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.252+0000 7efc5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:37.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.252+0000 7efc5d7fa640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:17:37.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.252+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc601d51f0 con 0x7efc601430c0 2024-03-25T18:17:37.259 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.258+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 589759192 0 0) 0x7efc44004eb0 con 0x7efc6013f710 2024-03-25T18:17:37.259 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.258+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc50003520 con 0x7efc6013f710 2024-03-25T18:17:37.262 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.261+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652312577 0 0) 0x7efc601379e0 con 0x7efc600a98f0 2024-03-25T18:17:37.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.261+0000 7efc5d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:37.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.261+0000 7efc5d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:37.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.261+0000 7efc5d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:37.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.261+0000 7efc5d7fa640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:17:37.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.261+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc44004eb0 con 0x7efc600a98f0 2024-03-25T18:17:37.269 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7efc4c001980 con 0x7efc6013f710 2024-03-25T18:17:37.269 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc5d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:37.269 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc5d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:37.269 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc5d7fa640 10 monclient(hunting): dump: 2024-03-25T18:17:37.269 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:37.270 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:37.270 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:37.270 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:37.270 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:37.270 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:37.270 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:37.270 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:37.270 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:37.270 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:37.270 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc5d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:37.270 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 460430410 0 0) 0x7efc50003520 con 0x7efc6013f710 2024-03-25T18:17:37.270 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 >> v1:172.21.15.122:6790/0 conn(0x7efc600a98f0 legacy=0x7efc60138230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.270 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 >> v1:172.21.15.122:6789/0 conn(0x7efc601430c0 legacy=0x7efc601db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.270 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc5d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:17:37.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc5d7fa640 1 monclient: found mon.b 2024-03-25T18:17:37.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc5d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:37.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc5d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:37.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc601db8b0 con 0x7efc6013f710 2024-03-25T18:17:37.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc5d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:17:37.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc5d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:17:37.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc5d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:37.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc66fef640 5 monclient: authenticate success, global_id 4276 2024-03-25T18:17:37.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc66fef640 10 monclient: _renew_subs 2024-03-25T18:17:37.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc66fef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:37.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc66fef640 1 -- 172.21.15.122:0/4185174047 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc601e2dd0 con 0x7efc6013f710 2024-03-25T18:17:37.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc66fef640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:37.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc66fef640 10 monclient: _renew_subs 2024-03-25T18:17:37.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc66fef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:37.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc66fef640 1 -- 172.21.15.122:0/4185174047 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc601e3140 con 0x7efc6013f710 2024-03-25T18:17:37.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.267+0000 7efc66fef640 1 librados: init done 2024-03-25T18:17:37.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.269+0000 7efc66fef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:37.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.269+0000 7efc66fef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:37.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.269+0000 7efc66fef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:37.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.269+0000 7efc66fef640 1 -- 172.21.15.122:0/4185174047 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc600b3ef0 con 0x7efc6013f710 2024-03-25T18:17:37.559 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.558+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc4c0035f0 con 0x7efc6013f710 2024-03-25T18:17:37.559 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.558+0000 7efc5d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:37.559 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.558+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7efc4c003a00 con 0x7efc6013f710 2024-03-25T18:17:37.559 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.558+0000 7efc5d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:37.559 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.558+0000 7efc5d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:37.559 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.558+0000 7efc5d7fa640 10 monclient: dump: 2024-03-25T18:17:37.559 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:37.559 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:37.559 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:37.560 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:37.560 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:37.560 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:37.560 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:37.560 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:37.560 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:37.560 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:37.560 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.558+0000 7efc5effd640 4 set_mon_vals no callback set 2024-03-25T18:17:37.561 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.559+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7efc4c003c20 con 0x7efc6013f710 2024-03-25T18:17:37.561 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.559+0000 7efc64d8b640 10 monclient: get_auth_request con 0x7efc4406de60 auth_method 0 2024-03-25T18:17:37.561 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.560+0000 7efc5d7fa640 10 client.4276.objecter ms_handle_connect 0x7efc4406de60 2024-03-25T18:17:37.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.564+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7efc4c0535e0 con 0x7efc6013f710 2024-03-25T18:17:37.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.564+0000 7efc5d7fa640 10 client.4276.objecter ms_dispatch 0x7efc60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:37.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.564+0000 7efc5d7fa640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:37.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.564+0000 7efc5d7fa640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:37.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.564+0000 7efc5d7fa640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-25T18:17:37.566 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.564+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc4c04f0f0 con 0x7efc6013f710 2024-03-25T18:17:37.566 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.564+0000 7efc5d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:37.566 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.564+0000 7efc5d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:37.802 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.801+0000 7efc66fef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:37.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.801+0000 7efc66fef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:37.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.801+0000 7efc66fef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:37.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.801+0000 7efc66fef640 1 -- 172.21.15.122:0/4185174047 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc600627b0 con 0x7efc6013f710 2024-03-25T18:17:37.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.832+0000 7efc5d7fa640 1 -- 172.21.15.122:0/4185174047 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7efc4c051020 con 0x7efc6013f710 2024-03-25T18:17:37.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.832+0000 7efc5d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:37.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.832+0000 7efc5d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:37.833 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:17:37.834 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:37.836 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.835+0000 7efc66fef640 10 librados: watch_flush enter 2024-03-25T18:17:37.836 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.835+0000 7efc66fef640 10 librados: watch_flush exit 2024-03-25T18:17:37.836 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.835+0000 7efc66fef640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-25T18:17:37.836 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.835+0000 7efc66fef640 10 client.4276.objecter successfully canceled tick 2024-03-25T18:17:37.836 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.835+0000 7efc66fef640 1 -- 172.21.15.122:0/4185174047 >> v1:172.21.15.204:6800/676228005 conn(0x7efc4406de60 legacy=0x7efc44070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.837 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.835+0000 7efc66fef640 10 monclient: shutdown 2024-03-25T18:17:37.837 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.835+0000 7efc66fef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:37.837 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.835+0000 7efc66fef640 1 -- 172.21.15.122:0/4185174047 >> v1:172.21.15.204:6789/0 conn(0x7efc6013f710 legacy=0x7efc601daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.837 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.835+0000 7efc66fef640 1 -- 172.21.15.122:0/4185174047 shutdown_connections 2024-03-25T18:17:37.837 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.835+0000 7efc66fef640 1 -- 172.21.15.122:0/4185174047 >> 172.21.15.122:0/4185174047 conn(0x7efc600b7ea0 msgr2=0x7efc60074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:37.837 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.835+0000 7efc66fef640 1 -- 172.21.15.122:0/4185174047 shutdown_connections 2024-03-25T18:17:37.837 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.835+0000 7efc66fef640 1 -- 172.21.15.122:0/4185174047 wait complete. 2024-03-25T18:17:37.837 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:37.835+0000 7efc66fef640 1 librados: shutdown 2024-03-25T18:17:37.848 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:44.653 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:44.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.753+0000 7f77cc939640 10 monclient: get_monmap_and_config 2024-03-25T18:17:44.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.753+0000 7f77cc939640 10 monclient: build_initial_monmap 2024-03-25T18:17:44.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.753+0000 7f77cc939640 10 monclient: monmap: 2024-03-25T18:17:44.754 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:17:44.754 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:44.754 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:17:44.754314+0000 2024-03-25T18:17:44.754 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:17:44.754314+0000 2024-03-25T18:17:44.754 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:44.754 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:44.754 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:17:44.755 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:17:44.755 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:17:44.755 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:44.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.753+0000 7f77cc939640 1 Processor -- start 2024-03-25T18:17:44.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.753+0000 7f77cc939640 1 -- start start 2024-03-25T18:17:44.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.753+0000 7f77cc939640 10 monclient: init 2024-03-25T18:17:44.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.754+0000 7f77cc939640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:44.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.754+0000 7f77cc939640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:44.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.754+0000 7f77cc939640 10 monclient(hunting): picked mon.noname-c con 0x7f77c80b4d60 addr v1:172.21.15.204:6789/0 2024-03-25T18:17:44.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.754+0000 7f77cc939640 10 monclient(hunting): picked mon.noname-a con 0x7f77c80b1730 addr v1:172.21.15.122:6789/0 2024-03-25T18:17:44.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.754+0000 7f77cc939640 10 monclient(hunting): picked mon.noname-b con 0x7f77c814be90 addr v1:172.21.15.122:6790/0 2024-03-25T18:17:44.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.754+0000 7f77cc939640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c80af940 con 0x7f77c80b1730 2024-03-25T18:17:44.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.754+0000 7f77cc939640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c80b0a70 con 0x7f77c80b4d60 2024-03-25T18:17:44.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.754+0000 7f77cc939640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c80b0d50 con 0x7f77c814be90 2024-03-25T18:17:44.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.754+0000 7f77cc939640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:44.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.754+0000 7f77cc939640 10 monclient(hunting): authenticate will time out at 7388.020629s 2024-03-25T18:17:44.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.755+0000 7f77c659c640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f77c80b4d60 0x7f77c813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:42218/0 (socket says 172.21.15.122:42218) 2024-03-25T18:17:44.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.755+0000 7f77c659c640 1 -- 172.21.15.122:0/4081754320 learned_addr learned my addr 172.21.15.122:0/4081754320 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:17:44.812 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.811+0000 7f77c559a640 1 -- 172.21.15.122:0/4081754320 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641178552 0 0) 0x7f77c80af940 con 0x7f77c80b1730 2024-03-25T18:17:44.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.812+0000 7f77c559a640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.812+0000 7f77c559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.812+0000 7f77c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.812+0000 7f77c559a640 10 monclient(hunting): my global_id is 4281 2024-03-25T18:17:44.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.812+0000 7f77c559a640 1 -- 172.21.15.122:0/4081754320 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77b0003520 con 0x7f77c80b1730 2024-03-25T18:17:44.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77c559a640 1 -- 172.21.15.122:0/4081754320 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4008730051 0 0) 0x7f77b0003520 con 0x7f77c80b1730 2024-03-25T18:17:44.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77c559a640 1 -- 172.21.15.122:0/4081754320 >> v1:172.21.15.122:6790/0 conn(0x7f77c814be90 legacy=0x7f77c814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77c559a640 1 -- 172.21.15.122:0/4081754320 >> v1:172.21.15.204:6789/0 conn(0x7f77c80b4d60 legacy=0x7f77c813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77c559a640 10 monclient: _finish_hunting 0 2024-03-25T18:17:44.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77c559a640 1 monclient: found mon.noname-a 2024-03-25T18:17:44.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:44.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77c559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:17:44.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77c559a640 1 -- 172.21.15.122:0/4081754320 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c80b1030 con 0x7f77c80b1730 2024-03-25T18:17:44.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77c559a640 10 monclient: _finish_auth 0 2024-03-25T18:17:44.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77c559a640 10 monclient: _check_auth_tickets 2024-03-25T18:17:44.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:44.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77c559a640 1 -- 172.21.15.122:0/4081754320 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f77b8001690 con 0x7f77c80b1730 2024-03-25T18:17:44.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77cc939640 5 monclient: authenticate success, global_id 4281 2024-03-25T18:17:44.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77cc939640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:44.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77c559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:44.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77c559a640 10 monclient: dump: 2024-03-25T18:17:44.826 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:44.826 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:44.827 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:44.827 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:44.827 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.827 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:44.827 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:44.827 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:44.827 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:44.827 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:44.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.824+0000 7f77cc939640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:44.840 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.839+0000 7f77c559a640 1 -- 172.21.15.122:0/4081754320 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77b80017f0 con 0x7f77c80b1730 2024-03-25T18:17:44.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.839+0000 7f77c559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:44.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.839+0000 7f77c559a640 1 -- 172.21.15.122:0/4081754320 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f77b8001c00 con 0x7f77c80b1730 2024-03-25T18:17:44.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.839+0000 7f77c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.839+0000 7f77c559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:44.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.839+0000 7f77c559a640 10 monclient: dump: 2024-03-25T18:17:44.841 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:44.841 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:44.841 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:44.841 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:44.841 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.841 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:44.841 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:44.841 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:44.841 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:44.842 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:44.842 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.840+0000 7f77cc939640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:44.842 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.840+0000 7f77cc939640 4 set_mon_vals no callback set 2024-03-25T18:17:44.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.840+0000 7f77cc939640 10 monclient: shutdown 2024-03-25T18:17:44.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.840+0000 7f77cc939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:44.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.840+0000 7f77cc939640 1 -- 172.21.15.122:0/4081754320 >> v1:172.21.15.122:6789/0 conn(0x7f77c80b1730 legacy=0x7f77c80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.840+0000 7f77cc939640 1 -- 172.21.15.122:0/4081754320 shutdown_connections 2024-03-25T18:17:44.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.840+0000 7f77cc939640 1 -- 172.21.15.122:0/4081754320 >> 172.21.15.122:0/4081754320 conn(0x7f77c80617a0 msgr2=0x7f77c8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:44.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.840+0000 7f77cc939640 1 -- 172.21.15.122:0/4081754320 shutdown_connections 2024-03-25T18:17:44.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.841+0000 7f77cc939640 1 -- 172.21.15.122:0/4081754320 wait complete. 2024-03-25T18:17:44.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.841+0000 7f77cc939640 10 monclient: build_initial_monmap 2024-03-25T18:17:44.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.841+0000 7f77cc939640 10 monclient: monmap: 2024-03-25T18:17:44.845 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:17:44.845 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:44.845 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:17:44.845 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:17:44.845 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:44.845 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:17:44.845 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:17:44.845 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:17:44.845 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:17:44.845 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:44.845 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.842+0000 7f77cc939640 1 librados: starting msgr at 2024-03-25T18:17:44.845 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.842+0000 7f77cc939640 1 librados: starting objecter 2024-03-25T18:17:44.845 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.842+0000 7f77cc939640 1 Processor -- start 2024-03-25T18:17:44.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.842+0000 7f77cc939640 1 -- start start 2024-03-25T18:17:44.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.842+0000 7f77cc939640 1 librados: setting wanted keys 2024-03-25T18:17:44.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.842+0000 7f77cc939640 1 librados: calling monclient init 2024-03-25T18:17:44.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.842+0000 7f77cc939640 10 monclient: init 2024-03-25T18:17:44.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.842+0000 7f77cc939640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:44.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.842+0000 7f77cc939640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:44.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.842+0000 7f77cc939640 10 monclient(hunting): picked mon.noname-a con 0x7f77c80b1730 addr v1:172.21.15.122:6789/0 2024-03-25T18:17:44.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.842+0000 7f77cc939640 10 monclient(hunting): picked mon.noname-b con 0x7f77c80b4d60 addr v1:172.21.15.204:6789/0 2024-03-25T18:17:44.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.842+0000 7f77cc939640 10 monclient(hunting): picked mon.noname-c con 0x7f77c814be90 addr v1:172.21.15.122:6790/0 2024-03-25T18:17:44.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.842+0000 7f77cc939640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c8140380 con 0x7f77c80b1730 2024-03-25T18:17:44.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.842+0000 7f77cc939640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c8141ca0 con 0x7f77c80b4d60 2024-03-25T18:17:44.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.842+0000 7f77cc939640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c8140860 con 0x7f77c814be90 2024-03-25T18:17:44.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.842+0000 7f77cc939640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:44.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.842+0000 7f77cc939640 10 monclient(hunting): authenticate will time out at 7388.108703s 2024-03-25T18:17:44.860 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.859+0000 7f77c6d9d640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f77c814be90 0x7f77c81e22e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:47804/0 (socket says 172.21.15.122:47804) 2024-03-25T18:17:44.860 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.859+0000 7f77c6d9d640 1 -- 172.21.15.122:0/3759146352 learned_addr learned my addr 172.21.15.122:0/3759146352 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:17:44.860 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.860+0000 7f77b6ffd640 10 client.?.objecter ms_handle_connect 0x7f77c814be90 2024-03-25T18:17:44.860 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.860+0000 7f77b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:44.862 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.861+0000 7f77b6ffd640 10 client.?.objecter ms_handle_connect 0x7f77c80b4d60 2024-03-25T18:17:44.862 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.861+0000 7f77b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:44.871 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.870+0000 7f77b6ffd640 10 client.?.objecter ms_handle_connect 0x7f77c80b1730 2024-03-25T18:17:44.871 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.870+0000 7f77b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:44.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.888+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2438049285 0 0) 0x7f77c8140380 con 0x7f77c80b1730 2024-03-25T18:17:44.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.888+0000 7f77b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.888+0000 7f77b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.888+0000 7f77b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.888+0000 7f77b6ffd640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:17:44.890 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.888+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77a8004eb0 con 0x7f77c80b1730 2024-03-25T18:17:44.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.946+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954482904 0 0) 0x7f77c8141ca0 con 0x7f77c80b4d60 2024-03-25T18:17:44.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.946+0000 7f77b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.946+0000 7f77b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.946+0000 7f77b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.946+0000 7f77b6ffd640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:17:44.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.946+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77c8140380 con 0x7f77c80b4d60 2024-03-25T18:17:44.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.958+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3182127984 0 0) 0x7f77c8140380 con 0x7f77c80b4d60 2024-03-25T18:17:44.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.958+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77c8141ca0 con 0x7f77c80b4d60 2024-03-25T18:17:44.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.958+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f77b8001b80 con 0x7f77c80b4d60 2024-03-25T18:17:44.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.958+0000 7f77b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.958+0000 7f77b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:44.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.958+0000 7f77b6ffd640 10 monclient(hunting): dump: 2024-03-25T18:17:44.960 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:44.960 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:44.960 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:44.960 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:44.960 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.960 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:44.960 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:44.960 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:44.960 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:44.960 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:44.961 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.958+0000 7f77b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:44.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.963+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305931341 0 0) 0x7f77c8140860 con 0x7f77c814be90 2024-03-25T18:17:44.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.963+0000 7f77b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.963+0000 7f77b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.963+0000 7f77b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.963+0000 7f77b6ffd640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:17:44.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.963+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77c8140380 con 0x7f77c814be90 2024-03-25T18:17:44.966 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.965+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 520828487 0 0) 0x7f77a8004eb0 con 0x7f77c80b1730 2024-03-25T18:17:44.966 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.965+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77c8140860 con 0x7f77c80b1730 2024-03-25T18:17:44.967 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.965+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f77bc0018a0 con 0x7f77c80b1730 2024-03-25T18:17:44.967 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.965+0000 7f77b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.967 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.965+0000 7f77b6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:44.967 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.965+0000 7f77b6ffd640 10 monclient(hunting): dump: 2024-03-25T18:17:44.967 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:44.967 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:44.967 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:44.967 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:44.967 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.967 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:44.967 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:44.967 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:44.967 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:44.967 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:44.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3857041919 0 0) 0x7f77c8141ca0 con 0x7f77c80b4d60 2024-03-25T18:17:44.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 >> v1:172.21.15.122:6790/0 conn(0x7f77c814be90 legacy=0x7f77c81e22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 >> v1:172.21.15.122:6789/0 conn(0x7f77c80b1730 legacy=0x7f77c81db010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77b6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:44.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77b6ffd640 1 monclient: found mon.b 2024-03-25T18:17:44.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:44.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77b6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:44.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c81423a0 con 0x7f77c80b4d60 2024-03-25T18:17:44.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77b6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:44.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77b6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:44.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:44.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77cc939640 5 monclient: authenticate success, global_id 4282 2024-03-25T18:17:44.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77cc939640 10 monclient: _renew_subs 2024-03-25T18:17:44.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77cc939640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:44.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77cc939640 1 -- 172.21.15.122:0/3759146352 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77c8138650 con 0x7f77c80b4d60 2024-03-25T18:17:44.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77cc939640 10 client.4282.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:44.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77cc939640 10 monclient: _renew_subs 2024-03-25T18:17:44.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77cc939640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:44.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77cc939640 1 -- 172.21.15.122:0/3759146352 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77c8140d00 con 0x7f77c80b4d60 2024-03-25T18:17:44.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.978+0000 7f77cc939640 1 librados: init done 2024-03-25T18:17:44.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.979+0000 7f77cc939640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:44.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.979+0000 7f77cc939640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:44.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.979+0000 7f77cc939640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:44.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:44.979+0000 7f77cc939640 1 -- 172.21.15.122:0/3759146352 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77c8140580 con 0x7f77c80b4d60 2024-03-25T18:17:45.259 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.257+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77b80038a0 con 0x7f77c80b4d60 2024-03-25T18:17:45.259 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.257+0000 7f77b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:45.259 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.257+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f77b8003a10 con 0x7f77c80b4d60 2024-03-25T18:17:45.259 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.257+0000 7f77b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:45.259 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.257+0000 7f77b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:45.259 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.257+0000 7f77b6ffd640 10 monclient: dump: 2024-03-25T18:17:45.259 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:45.259 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:45.259 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:45.260 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:45.260 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:45.260 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:45.260 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:45.260 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:45.260 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:45.260 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:45.260 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.257+0000 7f77c4d99640 4 set_mon_vals no callback set 2024-03-25T18:17:45.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.258+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f77b80199e0 con 0x7f77c80b4d60 2024-03-25T18:17:45.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.259+0000 7f77c659c640 10 monclient: get_auth_request con 0x7f77a806dfb0 auth_method 0 2024-03-25T18:17:45.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.259+0000 7f77b6ffd640 10 client.4282.objecter ms_handle_connect 0x7f77a806dfb0 2024-03-25T18:17:45.264 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.263+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f77b8053580 con 0x7f77c80b4d60 2024-03-25T18:17:45.264 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.263+0000 7f77b6ffd640 10 client.4282.objecter ms_dispatch 0x7f77c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:45.265 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.263+0000 7f77b6ffd640 3 client.4282.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:45.265 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.263+0000 7f77b6ffd640 3 client.4282.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:45.265 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.263+0000 7f77b6ffd640 20 client.4282.objecter dump_active .. 0 homeless 2024-03-25T18:17:45.265 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.264+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f77b804f090 con 0x7f77c80b4d60 2024-03-25T18:17:45.265 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.264+0000 7f77b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:45.265 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.264+0000 7f77b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:45.303 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:01:39.046 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.350 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:01:39.096 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.352 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:01:39.098 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.353 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:01:39.096 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.394 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:01:39.140 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.409 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:01:39.151 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.422 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:01:39.164 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.430 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:01:39.172 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.469+0000 7f77cc939640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:45.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.469+0000 7f77cc939640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:45.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.469+0000 7f77cc939640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:45.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.469+0000 7f77cc939640 1 -- 172.21.15.122:0/3759146352 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77c81389a0 con 0x7f77c80b4d60 2024-03-25T18:17:45.494 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:01:39.236 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:45.502 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.501+0000 7f77b6ffd640 1 -- 172.21.15.122:0/3759146352 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f77b8050fc0 con 0x7f77c80b4d60 2024-03-25T18:17:45.502 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.501+0000 7f77b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:45.502 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.501+0000 7f77b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:45.502 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:17:45.503 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:45.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.503+0000 7f77cc939640 10 librados: watch_flush enter 2024-03-25T18:17:45.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.503+0000 7f77cc939640 10 librados: watch_flush exit 2024-03-25T18:17:45.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.503+0000 7f77cc939640 20 client.4282.objecter shutdown clearing up homeless session... 2024-03-25T18:17:45.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.503+0000 7f77cc939640 10 client.4282.objecter successfully canceled tick 2024-03-25T18:17:45.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.503+0000 7f77cc939640 1 -- 172.21.15.122:0/3759146352 >> v1:172.21.15.204:6800/676228005 conn(0x7f77a806dfb0 legacy=0x7f77a8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:45.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.503+0000 7f77cc939640 10 monclient: shutdown 2024-03-25T18:17:45.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.503+0000 7f77cc939640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:45.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.503+0000 7f77cc939640 1 -- 172.21.15.122:0/3759146352 >> v1:172.21.15.204:6789/0 conn(0x7f77c80b4d60 legacy=0x7f77c81db720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:45.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.503+0000 7f77cc939640 1 -- 172.21.15.122:0/3759146352 shutdown_connections 2024-03-25T18:17:45.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.503+0000 7f77cc939640 1 -- 172.21.15.122:0/3759146352 >> 172.21.15.122:0/3759146352 conn(0x7f77c80617a0 msgr2=0x7f77c813c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:45.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.503+0000 7f77cc939640 1 -- 172.21.15.122:0/3759146352 shutdown_connections 2024-03-25T18:17:45.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.503+0000 7f77cc939640 1 -- 172.21.15.122:0/3759146352 wait complete. 2024-03-25T18:17:45.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:45.503+0000 7f77cc939640 1 librados: shutdown 2024-03-25T18:17:45.514 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:52.321 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:52.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.432+0000 7fa9eb6d9640 10 monclient: get_monmap_and_config 2024-03-25T18:17:52.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.432+0000 7fa9eb6d9640 10 monclient: build_initial_monmap 2024-03-25T18:17:52.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.432+0000 7fa9eb6d9640 10 monclient: monmap: 2024-03-25T18:17:52.433 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:17:52.433 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:52.433 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:17:52.433394+0000 2024-03-25T18:17:52.433 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:17:52.433394+0000 2024-03-25T18:17:52.433 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:52.434 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:52.434 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:17:52.434 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:17:52.434 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:17:52.434 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:52.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.433+0000 7fa9eb6d9640 1 Processor -- start 2024-03-25T18:17:52.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.433+0000 7fa9eb6d9640 1 -- start start 2024-03-25T18:17:52.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.433+0000 7fa9eb6d9640 10 monclient: init 2024-03-25T18:17:52.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.434+0000 7fa9eb6d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:52.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.434+0000 7fa9eb6d9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:52.436 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.434+0000 7fa9eb6d9640 10 monclient(hunting): picked mon.noname-b con 0x7fa9e40a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:17:52.436 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.434+0000 7fa9eb6d9640 10 monclient(hunting): picked mon.noname-c con 0x7fa9e413f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:17:52.436 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.434+0000 7fa9eb6d9640 10 monclient(hunting): picked mon.noname-a con 0x7fa9e41430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:17:52.436 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.434+0000 7fa9eb6d9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e4142520 con 0x7fa9e41430c0 2024-03-25T18:17:52.436 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.434+0000 7fa9eb6d9640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e40b28a0 con 0x7fa9e413f710 2024-03-25T18:17:52.436 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.434+0000 7fa9eb6d9640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e40b2a30 con 0x7fa9e40a98f0 2024-03-25T18:17:52.436 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.434+0000 7fa9eb6d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:52.436 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.434+0000 7fa9eb6d9640 10 monclient(hunting): authenticate will time out at 7395.700357s 2024-03-25T18:17:52.442 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.441+0000 7fa9e8c74640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fa9e413f710 0x7fa9e41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:40182/0 (socket says 172.21.15.122:40182) 2024-03-25T18:17:52.442 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.441+0000 7fa9e8c74640 1 -- 172.21.15.122:0/3583492359 learned_addr learned my addr 172.21.15.122:0/3583492359 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:17:52.451 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.449+0000 7fa9dbfff640 1 -- 172.21.15.122:0/3583492359 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935335988 0 0) 0x7fa9e40b28a0 con 0x7fa9e413f710 2024-03-25T18:17:52.451 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.449+0000 7fa9dbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:52.451 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.449+0000 7fa9dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:52.451 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.449+0000 7fa9dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:52.451 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.449+0000 7fa9dbfff640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:17:52.451 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.449+0000 7fa9dbfff640 1 -- 172.21.15.122:0/3583492359 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9d0003520 con 0x7fa9e413f710 2024-03-25T18:17:52.456 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.456+0000 7fa9dbfff640 1 -- 172.21.15.122:0/3583492359 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899661516 0 0) 0x7fa9e4142520 con 0x7fa9e41430c0 2024-03-25T18:17:52.457 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.456+0000 7fa9dbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:52.457 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.456+0000 7fa9dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:52.457 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.456+0000 7fa9dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:52.457 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.456+0000 7fa9dbfff640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:17:52.457 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.456+0000 7fa9dbfff640 1 -- 172.21.15.122:0/3583492359 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e40b28a0 con 0x7fa9e41430c0 2024-03-25T18:17:52.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.462+0000 7fa9dbfff640 1 -- 172.21.15.122:0/3583492359 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3005698247 0 0) 0x7fa9d0003520 con 0x7fa9e413f710 2024-03-25T18:17:52.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.462+0000 7fa9dbfff640 1 -- 172.21.15.122:0/3583492359 >> v1:172.21.15.122:6790/0 conn(0x7fa9e40a98f0 legacy=0x7fa9e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:17:52.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.462+0000 7fa9dbfff640 1 -- 172.21.15.122:0/3583492359 >> v1:172.21.15.122:6789/0 conn(0x7fa9e41430c0 legacy=0x7fa9e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:52.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.462+0000 7fa9dbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:52.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.462+0000 7fa9dbfff640 1 monclient: found mon.noname-c 2024-03-25T18:17:52.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.462+0000 7fa9dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:52.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.462+0000 7fa9dbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:17:52.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.462+0000 7fa9dbfff640 1 -- 172.21.15.122:0/3583492359 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e40b44c0 con 0x7fa9e413f710 2024-03-25T18:17:52.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.462+0000 7fa9dbfff640 10 monclient: _finish_auth 0 2024-03-25T18:17:52.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.462+0000 7fa9dbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:52.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.462+0000 7fa9dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:52.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.462+0000 7fa9dbfff640 1 -- 172.21.15.122:0/3583492359 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fa9d4001690 con 0x7fa9e413f710 2024-03-25T18:17:52.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.462+0000 7fa9dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:52.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.463+0000 7fa9dbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:52.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.463+0000 7fa9dbfff640 10 monclient: dump: 2024-03-25T18:17:52.465 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:52.465 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:52.465 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:52.465 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:52.465 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:52.465 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:52.465 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:52.465 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:52.465 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:52.465 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:52.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.463+0000 7fa9eb6d9640 5 monclient: authenticate success, global_id 4285 2024-03-25T18:17:52.466 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.463+0000 7fa9eb6d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:52.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.472+0000 7fa9dbfff640 1 -- 172.21.15.122:0/3583492359 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9d40017f0 con 0x7fa9e413f710 2024-03-25T18:17:52.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.472+0000 7fa9dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:52.474 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.472+0000 7fa9eb6d9640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:52.474 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.472+0000 7fa9eb6d9640 4 set_mon_vals no callback set 2024-03-25T18:17:52.474 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.472+0000 7fa9dbfff640 1 -- 172.21.15.122:0/3583492359 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fa9d4001c00 con 0x7fa9e413f710 2024-03-25T18:17:52.474 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.472+0000 7fa9dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:52.474 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.472+0000 7fa9dbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:52.474 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.472+0000 7fa9dbfff640 10 monclient: dump: 2024-03-25T18:17:52.474 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:52.474 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:52.474 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:52.474 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:52.474 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:52.474 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:52.474 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:52.474 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:52.475 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:52.475 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:52.476 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.473+0000 7fa9eb6d9640 10 monclient: shutdown 2024-03-25T18:17:52.476 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.473+0000 7fa9eb6d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:52.476 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.473+0000 7fa9eb6d9640 1 -- 172.21.15.122:0/3583492359 >> v1:172.21.15.204:6789/0 conn(0x7fa9e413f710 legacy=0x7fa9e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:52.476 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.473+0000 7fa9eb6d9640 1 -- 172.21.15.122:0/3583492359 shutdown_connections 2024-03-25T18:17:52.476 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.473+0000 7fa9eb6d9640 1 -- 172.21.15.122:0/3583492359 >> 172.21.15.122:0/3583492359 conn(0x7fa9e40b7ea0 msgr2=0x7fa9e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:52.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.473+0000 7fa9eb6d9640 1 -- 172.21.15.122:0/3583492359 shutdown_connections 2024-03-25T18:17:52.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.473+0000 7fa9eb6d9640 1 -- 172.21.15.122:0/3583492359 wait complete. 2024-03-25T18:17:52.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.474+0000 7fa9eb6d9640 10 monclient: build_initial_monmap 2024-03-25T18:17:52.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.474+0000 7fa9eb6d9640 10 monclient: monmap: 2024-03-25T18:17:52.477 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:17:52.477 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:52.477 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:17:52.477 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:17:52.477 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:52.477 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:17:52.477 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:17:52.477 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:17:52.477 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:17:52.477 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:52.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.475+0000 7fa9eb6d9640 1 librados: starting msgr at 2024-03-25T18:17:52.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.475+0000 7fa9eb6d9640 1 librados: starting objecter 2024-03-25T18:17:52.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.475+0000 7fa9eb6d9640 1 Processor -- start 2024-03-25T18:17:52.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.475+0000 7fa9eb6d9640 1 -- start start 2024-03-25T18:17:52.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.475+0000 7fa9eb6d9640 1 librados: setting wanted keys 2024-03-25T18:17:52.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.475+0000 7fa9eb6d9640 1 librados: calling monclient init 2024-03-25T18:17:52.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.475+0000 7fa9eb6d9640 10 monclient: init 2024-03-25T18:17:52.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.476+0000 7fa9eb6d9640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:52.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.476+0000 7fa9eb6d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:52.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.476+0000 7fa9eb6d9640 10 monclient(hunting): picked mon.noname-a con 0x7fa9e40a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:17:52.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.476+0000 7fa9eb6d9640 10 monclient(hunting): picked mon.noname-c con 0x7fa9e413f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:17:52.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.476+0000 7fa9eb6d9640 10 monclient(hunting): picked mon.noname-b con 0x7fa9e41430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:17:52.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.476+0000 7fa9eb6d9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e4139510 con 0x7fa9e40a98f0 2024-03-25T18:17:52.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.476+0000 7fa9eb6d9640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e41366a0 con 0x7fa9e41430c0 2024-03-25T18:17:52.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.476+0000 7fa9eb6d9640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9e41d7e80 con 0x7fa9e413f710 2024-03-25T18:17:52.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.476+0000 7fa9eb6d9640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:52.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.476+0000 7fa9eb6d9640 10 monclient(hunting): authenticate will time out at 7395.742185s 2024-03-25T18:17:52.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.487+0000 7fa9e9c76640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fa9e41430c0 0x7fa9e41372d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:40222/0 (socket says 172.21.15.122:40222) 2024-03-25T18:17:52.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.487+0000 7fa9e9c76640 1 -- 172.21.15.122:0/2612193575 learned_addr learned my addr 172.21.15.122:0/2612193575 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:17:52.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.487+0000 7fa9e9475640 1 --1- 172.21.15.122:0/2612193575 >> v1:172.21.15.122:6789/0 conn(0x7fa9e40a98f0 0x7fa9e41d7770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37366/0 (socket says 172.21.15.122:37366) 2024-03-25T18:17:52.490 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.489+0000 7fa9d9ffb640 10 client.?.objecter ms_handle_connect 0x7fa9e41430c0 2024-03-25T18:17:52.490 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.489+0000 7fa9d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:52.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.490+0000 7fa9d9ffb640 10 client.?.objecter ms_handle_connect 0x7fa9e40a98f0 2024-03-25T18:17:52.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.490+0000 7fa9d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:52.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.492+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643774305 0 0) 0x7fa9e41366a0 con 0x7fa9e41430c0 2024-03-25T18:17:52.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.492+0000 7fa9d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:52.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.492+0000 7fa9d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:52.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.492+0000 7fa9d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:52.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.492+0000 7fa9d9ffb640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:17:52.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.492+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c8004eb0 con 0x7fa9e41430c0 2024-03-25T18:17:52.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.496+0000 7fa9d9ffb640 10 client.?.objecter ms_handle_connect 0x7fa9e413f710 2024-03-25T18:17:52.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.496+0000 7fa9d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:52.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.505+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1571701705 0 0) 0x7fa9c8004eb0 con 0x7fa9e41430c0 2024-03-25T18:17:52.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.505+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9e41366a0 con 0x7fa9e41430c0 2024-03-25T18:17:52.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.505+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fa9dc001c70 con 0x7fa9e41430c0 2024-03-25T18:17:52.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.505+0000 7fa9d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:52.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.505+0000 7fa9d9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:52.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.505+0000 7fa9d9ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:52.506 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:52.506 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:52.506 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:52.507 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:52.507 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:52.507 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:52.507 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:52.507 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:52.507 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:52.507 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:52.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.505+0000 7fa9d9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:52.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.506+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2611280275 0 0) 0x7fa9e41d7e80 con 0x7fa9e413f710 2024-03-25T18:17:52.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.506+0000 7fa9d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:52.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.506+0000 7fa9d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:52.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.506+0000 7fa9d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:52.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.506+0000 7fa9d9ffb640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:17:52.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.506+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9c8004eb0 con 0x7fa9e413f710 2024-03-25T18:17:52.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.508+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 265604390 0 0) 0x7fa9e4139510 con 0x7fa9e40a98f0 2024-03-25T18:17:52.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.508+0000 7fa9d9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:52.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.508+0000 7fa9d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:52.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.508+0000 7fa9d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:52.510 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.508+0000 7fa9d9ffb640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:17:52.510 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.508+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e41d7e80 con 0x7fa9e40a98f0 2024-03-25T18:17:52.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.518+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4292147569 0 0) 0x7fa9c8004eb0 con 0x7fa9e413f710 2024-03-25T18:17:52.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.518+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9e4139510 con 0x7fa9e413f710 2024-03-25T18:17:52.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.518+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fa9d4001b00 con 0x7fa9e413f710 2024-03-25T18:17:52.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.518+0000 7fa9d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:52.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.518+0000 7fa9d9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:52.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.518+0000 7fa9d9ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:52.519 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:52.519 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:52.520 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:52.520 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:52.520 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:52.520 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:52.520 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:52.520 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:52.520 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:52.520 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:52.529 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.528+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2697020500 0 0) 0x7fa9e41d7e80 con 0x7fa9e40a98f0 2024-03-25T18:17:52.529 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.528+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9c8004eb0 con 0x7fa9e40a98f0 2024-03-25T18:17:52.529 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.528+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fa9e00017e0 con 0x7fa9e40a98f0 2024-03-25T18:17:52.529 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.528+0000 7fa9d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:52.530 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.528+0000 7fa9d9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:52.530 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.528+0000 7fa9d9ffb640 10 monclient(hunting): dump: 2024-03-25T18:17:52.530 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:52.530 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:52.530 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:52.530 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:52.530 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:52.530 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:52.530 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:52.530 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:52.530 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:52.530 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:52.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2973117187 0 0) 0x7fa9e41366a0 con 0x7fa9e41430c0 2024-03-25T18:17:52.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 >> v1:172.21.15.122:6790/0 conn(0x7fa9e413f710 legacy=0x7fa9e40b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:52.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 >> v1:172.21.15.122:6789/0 conn(0x7fa9e40a98f0 legacy=0x7fa9e41d7770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:52.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9d9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:52.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9d9ffb640 1 monclient: found mon.b 2024-03-25T18:17:52.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:52.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9d9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:52.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9e41d8010 con 0x7fa9e41430c0 2024-03-25T18:17:52.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9d9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:52.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9d9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:52.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:52.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9eb6d9640 5 monclient: authenticate success, global_id 4288 2024-03-25T18:17:52.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9eb6d9640 10 monclient: _renew_subs 2024-03-25T18:17:52.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9eb6d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:52.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9eb6d9640 1 -- 172.21.15.122:0/2612193575 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9e414e5e0 con 0x7fa9e41430c0 2024-03-25T18:17:52.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9eb6d9640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:52.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9eb6d9640 10 monclient: _renew_subs 2024-03-25T18:17:52.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9eb6d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:52.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9eb6d9640 1 -- 172.21.15.122:0/2612193575 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9e40b14b0 con 0x7fa9e41430c0 2024-03-25T18:17:52.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.531+0000 7fa9eb6d9640 1 librados: init done 2024-03-25T18:17:52.534 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.532+0000 7fa9eb6d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:52.534 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.532+0000 7fa9eb6d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:52.534 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.532+0000 7fa9eb6d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:52.534 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.532+0000 7fa9eb6d9640 1 -- 172.21.15.122:0/2612193575 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9e41d8440 con 0x7fa9e41430c0 2024-03-25T18:17:52.812 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.811+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9dc003990 con 0x7fa9e41430c0 2024-03-25T18:17:52.812 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.811+0000 7fa9d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:52.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.811+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fa9dc003da0 con 0x7fa9e41430c0 2024-03-25T18:17:52.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.811+0000 7fa9d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:52.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.811+0000 7fa9d9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:52.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.811+0000 7fa9d9ffb640 10 monclient: dump: 2024-03-25T18:17:52.813 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:17:52.813 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:17:52.813 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:52.813 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:17:52.813 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:52.813 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:17:52.813 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:17:52.813 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:17:52.813 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:17:52.813 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:17:52.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.811+0000 7fa9db7fe640 4 set_mon_vals no callback set 2024-03-25T18:17:52.814 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.812+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fa9dc003fc0 con 0x7fa9e41430c0 2024-03-25T18:17:52.814 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.813+0000 7fa9e9475640 10 monclient: get_auth_request con 0x7fa9c806e1d0 auth_method 0 2024-03-25T18:17:52.814 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.813+0000 7fa9d9ffb640 10 client.4288.objecter ms_handle_connect 0x7fa9c806e1d0 2024-03-25T18:17:52.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.818+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fa9dc054ad0 con 0x7fa9e41430c0 2024-03-25T18:17:52.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.818+0000 7fa9d9ffb640 10 client.4288.objecter ms_dispatch 0x7fa9e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:52.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.818+0000 7fa9d9ffb640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:52.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.818+0000 7fa9d9ffb640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:52.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.818+0000 7fa9d9ffb640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T18:17:52.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.819+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9dc0505e0 con 0x7fa9e41430c0 2024-03-25T18:17:52.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.819+0000 7fa9d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:52.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:52.819+0000 7fa9d9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:53.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.026+0000 7fa9eb6d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:53.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.026+0000 7fa9eb6d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:53.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.026+0000 7fa9eb6d9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:17:53.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.026+0000 7fa9eb6d9640 1 -- 172.21.15.122:0/2612193575 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9e40627b0 con 0x7fa9e41430c0 2024-03-25T18:17:53.114 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.113+0000 7fa9d9ffb640 1 -- 172.21.15.122:0/2612193575 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fa9dc052510 con 0x7fa9e41430c0 2024-03-25T18:17:53.114 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.113+0000 7fa9d9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:53.114 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.113+0000 7fa9d9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:53.115 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:17:53.115 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:53.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.116+0000 7fa9eb6d9640 10 librados: watch_flush enter 2024-03-25T18:17:53.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.116+0000 7fa9eb6d9640 10 librados: watch_flush exit 2024-03-25T18:17:53.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.116+0000 7fa9eb6d9640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T18:17:53.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.116+0000 7fa9eb6d9640 10 client.4288.objecter successfully canceled tick 2024-03-25T18:17:53.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.116+0000 7fa9eb6d9640 1 -- 172.21.15.122:0/2612193575 >> v1:172.21.15.204:6800/676228005 conn(0x7fa9c806e1d0 legacy=0x7fa9c8070690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:53.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.116+0000 7fa9eb6d9640 10 monclient: shutdown 2024-03-25T18:17:53.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.116+0000 7fa9eb6d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:53.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.116+0000 7fa9eb6d9640 1 -- 172.21.15.122:0/2612193575 >> v1:172.21.15.204:6789/0 conn(0x7fa9e41430c0 legacy=0x7fa9e41372d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:53.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.116+0000 7fa9eb6d9640 1 -- 172.21.15.122:0/2612193575 shutdown_connections 2024-03-25T18:17:53.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.116+0000 7fa9eb6d9640 1 -- 172.21.15.122:0/2612193575 >> 172.21.15.122:0/2612193575 conn(0x7fa9e40b7ea0 msgr2=0x7fa9e413d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:53.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.116+0000 7fa9eb6d9640 1 -- 172.21.15.122:0/2612193575 shutdown_connections 2024-03-25T18:17:53.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.117+0000 7fa9eb6d9640 1 -- 172.21.15.122:0/2612193575 wait complete. 2024-03-25T18:17:53.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:17:53.117+0000 7fa9eb6d9640 1 librados: shutdown 2024-03-25T18:17:53.129 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:59.936 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:00.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.050+0000 7fd8c82c0640 10 monclient: get_monmap_and_config 2024-03-25T18:18:00.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.050+0000 7fd8c82c0640 10 monclient: build_initial_monmap 2024-03-25T18:18:00.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.050+0000 7fd8c82c0640 10 monclient: monmap: 2024-03-25T18:18:00.052 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:18:00.052 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:00.052 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:18:00.051943+0000 2024-03-25T18:18:00.052 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:18:00.051943+0000 2024-03-25T18:18:00.052 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:00.052 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:00.052 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:18:00.052 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:18:00.053 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:18:00.053 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:00.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.051+0000 7fd8c82c0640 1 Processor -- start 2024-03-25T18:18:00.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.051+0000 7fd8c82c0640 1 -- start start 2024-03-25T18:18:00.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.051+0000 7fd8c82c0640 10 monclient: init 2024-03-25T18:18:00.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.052+0000 7fd8c82c0640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:00.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.052+0000 7fd8c82c0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:00.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.052+0000 7fd8c82c0640 10 monclient(hunting): picked mon.noname-b con 0x7fd8c00a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:18:00.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.052+0000 7fd8c82c0640 10 monclient(hunting): picked mon.noname-c con 0x7fd8c013f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:18:00.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.052+0000 7fd8c82c0640 10 monclient(hunting): picked mon.noname-a con 0x7fd8c01430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:18:00.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.052+0000 7fd8c82c0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c0142520 con 0x7fd8c01430c0 2024-03-25T18:18:00.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.052+0000 7fd8c82c0640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c00b28a0 con 0x7fd8c013f710 2024-03-25T18:18:00.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.052+0000 7fd8c82c0640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c00b2a30 con 0x7fd8c00a98f0 2024-03-25T18:18:00.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.052+0000 7fd8c82c0640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:00.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.052+0000 7fd8c82c0640 10 monclient(hunting): authenticate will time out at 7403.318837s 2024-03-25T18:18:00.061 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.060+0000 7fd8c585b640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fd8c013f710 0x7fd8c01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:53274/0 (socket says 172.21.15.122:53274) 2024-03-25T18:18:00.061 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.060+0000 7fd8c585b640 1 -- 172.21.15.122:0/4098845253 learned_addr learned my addr 172.21.15.122:0/4098845253 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:18:00.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.072+0000 7fd8c505a640 1 -- 172.21.15.122:0/4098845253 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216196576 0 0) 0x7fd8c00b28a0 con 0x7fd8c013f710 2024-03-25T18:18:00.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.072+0000 7fd8c505a640 10 monclient(hunting): _negotiate 2024-03-25T18:18:00.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.072+0000 7fd8c505a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:00.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.072+0000 7fd8c505a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:00.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.072+0000 7fd8c505a640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:18:00.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.072+0000 7fd8c505a640 1 -- 172.21.15.122:0/4098845253 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8ac003520 con 0x7fd8c013f710 2024-03-25T18:18:00.076 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.075+0000 7fd8c505a640 1 -- 172.21.15.122:0/4098845253 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2551440330 0 0) 0x7fd8c00b2a30 con 0x7fd8c00a98f0 2024-03-25T18:18:00.076 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.075+0000 7fd8c505a640 10 monclient(hunting): _negotiate 2024-03-25T18:18:00.076 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.075+0000 7fd8c505a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:00.076 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.075+0000 7fd8c505a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:00.076 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.075+0000 7fd8c505a640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:18:00.076 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.075+0000 7fd8c505a640 1 -- 172.21.15.122:0/4098845253 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8c00b28a0 con 0x7fd8c00a98f0 2024-03-25T18:18:00.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.085+0000 7fd8c505a640 1 -- 172.21.15.122:0/4098845253 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 611893549 0 0) 0x7fd8c0142520 con 0x7fd8c01430c0 2024-03-25T18:18:00.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.085+0000 7fd8c505a640 10 monclient(hunting): _negotiate 2024-03-25T18:18:00.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.085+0000 7fd8c505a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:00.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.085+0000 7fd8c505a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:00.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.085+0000 7fd8c505a640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:18:00.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.085+0000 7fd8c505a640 1 -- 172.21.15.122:0/4098845253 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8c00b2a30 con 0x7fd8c01430c0 2024-03-25T18:18:00.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.097+0000 7fd8c505a640 1 -- 172.21.15.122:0/4098845253 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1501739784 0 0) 0x7fd8c00b28a0 con 0x7fd8c00a98f0 2024-03-25T18:18:00.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.097+0000 7fd8c505a640 1 -- 172.21.15.122:0/4098845253 >> v1:172.21.15.204:6789/0 conn(0x7fd8c013f710 legacy=0x7fd8c01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:00.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.097+0000 7fd8c505a640 1 -- 172.21.15.122:0/4098845253 >> v1:172.21.15.122:6789/0 conn(0x7fd8c01430c0 legacy=0x7fd8c014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:00.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.097+0000 7fd8c505a640 10 monclient: _finish_hunting 0 2024-03-25T18:18:00.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.097+0000 7fd8c505a640 1 monclient: found mon.noname-b 2024-03-25T18:18:00.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.097+0000 7fd8c505a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:00.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.097+0000 7fd8c505a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:18:00.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.097+0000 7fd8c505a640 1 -- 172.21.15.122:0/4098845253 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c00b44c0 con 0x7fd8c00a98f0 2024-03-25T18:18:00.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.097+0000 7fd8c505a640 10 monclient: _finish_auth 0 2024-03-25T18:18:00.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.097+0000 7fd8c505a640 10 monclient: _check_auth_tickets 2024-03-25T18:18:00.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.097+0000 7fd8c505a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:00.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.097+0000 7fd8c505a640 1 -- 172.21.15.122:0/4098845253 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd8bc001670 con 0x7fd8c00a98f0 2024-03-25T18:18:00.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.097+0000 7fd8c82c0640 5 monclient: authenticate success, global_id 4274 2024-03-25T18:18:00.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.097+0000 7fd8c82c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:00.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.098+0000 7fd8c505a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:00.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.098+0000 7fd8c505a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:00.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.098+0000 7fd8c505a640 10 monclient: dump: 2024-03-25T18:18:00.100 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:00.100 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:00.100 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:00.100 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:00.100 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:00.100 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:00.100 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:00.100 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:00.100 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:00.100 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:00.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.098+0000 7fd8c82c0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:00.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.117+0000 7fd8c505a640 1 -- 172.21.15.122:0/4098845253 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8bc0017d0 con 0x7fd8c00a98f0 2024-03-25T18:18:00.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.117+0000 7fd8c505a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:00.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.117+0000 7fd8c505a640 1 -- 172.21.15.122:0/4098845253 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd8bc001be0 con 0x7fd8c00a98f0 2024-03-25T18:18:00.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.117+0000 7fd8c82c0640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:00.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.117+0000 7fd8c82c0640 4 set_mon_vals no callback set 2024-03-25T18:18:00.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.117+0000 7fd8c505a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:00.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.117+0000 7fd8c505a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:00.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.117+0000 7fd8c505a640 10 monclient: dump: 2024-03-25T18:18:00.119 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:00.119 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:00.119 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:00.119 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:00.119 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:00.119 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:00.119 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:00.119 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:00.119 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:00.120 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:00.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.118+0000 7fd8c82c0640 10 monclient: shutdown 2024-03-25T18:18:00.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.118+0000 7fd8c82c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:00.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.118+0000 7fd8c82c0640 1 -- 172.21.15.122:0/4098845253 >> v1:172.21.15.122:6790/0 conn(0x7fd8c00a98f0 legacy=0x7fd8c00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:00.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.118+0000 7fd8c82c0640 1 -- 172.21.15.122:0/4098845253 shutdown_connections 2024-03-25T18:18:00.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.118+0000 7fd8c82c0640 1 -- 172.21.15.122:0/4098845253 >> 172.21.15.122:0/4098845253 conn(0x7fd8c00b7ea0 msgr2=0x7fd8c00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:00.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.118+0000 7fd8c82c0640 1 -- 172.21.15.122:0/4098845253 shutdown_connections 2024-03-25T18:18:00.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.118+0000 7fd8c82c0640 1 -- 172.21.15.122:0/4098845253 wait complete. 2024-03-25T18:18:00.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.119+0000 7fd8c82c0640 10 monclient: build_initial_monmap 2024-03-25T18:18:00.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.119+0000 7fd8c82c0640 10 monclient: monmap: 2024-03-25T18:18:00.121 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:18:00.121 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:00.121 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:18:00.122 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:18:00.122 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:00.122 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:18:00.122 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:18:00.122 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:18:00.122 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:18:00.122 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:00.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.120+0000 7fd8c82c0640 1 librados: starting msgr at 2024-03-25T18:18:00.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.120+0000 7fd8c82c0640 1 librados: starting objecter 2024-03-25T18:18:00.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.120+0000 7fd8c82c0640 1 Processor -- start 2024-03-25T18:18:00.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.120+0000 7fd8c82c0640 1 -- start start 2024-03-25T18:18:00.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.120+0000 7fd8c82c0640 1 librados: setting wanted keys 2024-03-25T18:18:00.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.120+0000 7fd8c82c0640 1 librados: calling monclient init 2024-03-25T18:18:00.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.120+0000 7fd8c82c0640 10 monclient: init 2024-03-25T18:18:00.123 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.121+0000 7fd8c82c0640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:00.124 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.121+0000 7fd8c82c0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:00.124 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.121+0000 7fd8c82c0640 10 monclient(hunting): picked mon.noname-b con 0x7fd8c00a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:18:00.124 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.121+0000 7fd8c82c0640 10 monclient(hunting): picked mon.noname-c con 0x7fd8c013f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:18:00.124 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.121+0000 7fd8c82c0640 10 monclient(hunting): picked mon.noname-a con 0x7fd8c01430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:18:00.124 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.121+0000 7fd8c82c0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c00abb90 con 0x7fd8c01430c0 2024-03-25T18:18:00.124 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.121+0000 7fd8c82c0640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c00b1840 con 0x7fd8c00a98f0 2024-03-25T18:18:00.124 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.121+0000 7fd8c82c0640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8c00acbf0 con 0x7fd8c013f710 2024-03-25T18:18:00.124 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.121+0000 7fd8c82c0640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:00.124 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.121+0000 7fd8c82c0640 10 monclient(hunting): authenticate will time out at 7403.387386s 2024-03-25T18:18:00.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.126+0000 7fd8c685d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd8c01430c0 0x7fd8c01e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38236/0 (socket says 172.21.15.122:38236) 2024-03-25T18:18:00.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.126+0000 7fd8c685d640 1 -- 172.21.15.122:0/4194859386 learned_addr learned my addr 172.21.15.122:0/4194859386 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:18:00.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.130+0000 7fd8b6ffd640 10 client.?.objecter ms_handle_connect 0x7fd8c01430c0 2024-03-25T18:18:00.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.130+0000 7fd8b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:00.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.132+0000 7fd8b6ffd640 10 client.?.objecter ms_handle_connect 0x7fd8c00a98f0 2024-03-25T18:18:00.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.132+0000 7fd8b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:00.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.141+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1262110911 0 0) 0x7fd8c00abb90 con 0x7fd8c01430c0 2024-03-25T18:18:00.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.141+0000 7fd8b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:00.143 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.141+0000 7fd8b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:00.143 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.141+0000 7fd8b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:00.143 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.141+0000 7fd8b6ffd640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:18:00.143 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.141+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8a4004eb0 con 0x7fd8c01430c0 2024-03-25T18:18:00.147 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.146+0000 7fd8b6ffd640 10 client.?.objecter ms_handle_connect 0x7fd8c013f710 2024-03-25T18:18:00.147 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.146+0000 7fd8b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:00.155 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.154+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1456894873 0 0) 0x7fd8c00acbf0 con 0x7fd8c013f710 2024-03-25T18:18:00.155 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.154+0000 7fd8b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:00.155 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.154+0000 7fd8b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:00.155 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.154+0000 7fd8b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:00.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.154+0000 7fd8b6ffd640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:18:00.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.154+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8c00abb90 con 0x7fd8c013f710 2024-03-25T18:18:00.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.154+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4072052310 0 0) 0x7fd8a4004eb0 con 0x7fd8c01430c0 2024-03-25T18:18:00.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.154+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8c00acbf0 con 0x7fd8c01430c0 2024-03-25T18:18:00.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.154+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd8b8001c70 con 0x7fd8c01430c0 2024-03-25T18:18:00.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.154+0000 7fd8b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:00.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.154+0000 7fd8b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:00.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.154+0000 7fd8b6ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:00.156 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:00.156 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:00.156 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:00.156 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:00.156 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:00.156 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:00.156 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:00.156 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:00.157 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:00.157 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:00.157 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.154+0000 7fd8b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:00.159 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.159+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016615570 0 0) 0x7fd8c00b1840 con 0x7fd8c00a98f0 2024-03-25T18:18:00.160 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.159+0000 7fd8b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:00.160 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.159+0000 7fd8b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:00.160 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.159+0000 7fd8b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:00.160 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.159+0000 7fd8b6ffd640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:18:00.160 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.159+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8a4004eb0 con 0x7fd8c00a98f0 2024-03-25T18:18:00.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1966465670 0 0) 0x7fd8c00acbf0 con 0x7fd8c01430c0 2024-03-25T18:18:00.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 >> v1:172.21.15.122:6790/0 conn(0x7fd8c013f710 legacy=0x7fd8c00ac4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:00.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 >> v1:172.21.15.204:6789/0 conn(0x7fd8c00a98f0 legacy=0x7fd8c0138860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:00.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8b6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:00.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8b6ffd640 1 monclient: found mon.a 2024-03-25T18:18:00.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:00.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8b6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:18:00.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c00abde0 con 0x7fd8c01430c0 2024-03-25T18:18:00.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8b6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:00.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8b6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:00.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:00.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8c82c0640 5 monclient: authenticate success, global_id 4296 2024-03-25T18:18:00.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8c82c0640 10 monclient: _renew_subs 2024-03-25T18:18:00.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8c82c0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:18:00.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8c82c0640 1 -- 172.21.15.122:0/4194859386 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8c00ac0c0 con 0x7fd8c01430c0 2024-03-25T18:18:00.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8c82c0640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:00.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8c82c0640 10 monclient: _renew_subs 2024-03-25T18:18:00.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8c82c0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:18:00.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8c82c0640 1 -- 172.21.15.122:0/4194859386 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8c0136580 con 0x7fd8c01430c0 2024-03-25T18:18:00.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.163+0000 7fd8c82c0640 1 librados: init done 2024-03-25T18:18:00.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.164+0000 7fd8c82c0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:00.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.164+0000 7fd8c82c0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:00.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.164+0000 7fd8c82c0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:18:00.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.164+0000 7fd8c82c0640 1 -- 172.21.15.122:0/4194859386 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8c00627b0 con 0x7fd8c01430c0 2024-03-25T18:18:00.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.470+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8b8003990 con 0x7fd8c01430c0 2024-03-25T18:18:00.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.470+0000 7fd8b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:00.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.470+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd8b8003da0 con 0x7fd8c01430c0 2024-03-25T18:18:00.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.470+0000 7fd8b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:00.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.470+0000 7fd8b6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:00.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.471+0000 7fd8b6ffd640 10 monclient: dump: 2024-03-25T18:18:00.472 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:00.472 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:00.472 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:00.472 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:00.473 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:00.473 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:00.473 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:00.473 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:00.473 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:00.473 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:00.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.470+0000 7fd8c4859640 4 set_mon_vals no callback set 2024-03-25T18:18:00.474 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.471+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fd8b8003fc0 con 0x7fd8c01430c0 2024-03-25T18:18:00.474 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.472+0000 7fd8c605c640 10 monclient: get_auth_request con 0x7fd8a406de00 auth_method 0 2024-03-25T18:18:00.474 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.472+0000 7fd8b6ffd640 10 client.4296.objecter ms_handle_connect 0x7fd8a406de00 2024-03-25T18:18:00.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.480+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fd8b8054ae0 con 0x7fd8c01430c0 2024-03-25T18:18:00.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.480+0000 7fd8b6ffd640 10 client.4296.objecter ms_dispatch 0x7fd8c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:00.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.480+0000 7fd8b6ffd640 3 client.4296.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:00.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.480+0000 7fd8b6ffd640 3 client.4296.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:00.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.480+0000 7fd8b6ffd640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-25T18:18:00.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.480+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd8b80505f0 con 0x7fd8c01430c0 2024-03-25T18:18:00.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.480+0000 7fd8b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:00.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.480+0000 7fd8b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:00.718 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.717+0000 7fd8c82c0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:00.718 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.717+0000 7fd8c82c0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:00.718 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.717+0000 7fd8c82c0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:18:00.718 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.717+0000 7fd8c82c0640 1 -- 172.21.15.122:0/4194859386 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8c013e150 con 0x7fd8c01430c0 2024-03-25T18:18:00.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.756+0000 7fd8b6ffd640 1 -- 172.21.15.122:0/4194859386 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fd8b8052520 con 0x7fd8c01430c0 2024-03-25T18:18:00.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.756+0000 7fd8b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:00.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.756+0000 7fd8b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:00.757 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:18:00.757 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:00.760 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.758+0000 7fd8c82c0640 10 librados: watch_flush enter 2024-03-25T18:18:00.760 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.758+0000 7fd8c82c0640 10 librados: watch_flush exit 2024-03-25T18:18:00.760 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.758+0000 7fd8c82c0640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-25T18:18:00.760 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.758+0000 7fd8c82c0640 10 client.4296.objecter successfully canceled tick 2024-03-25T18:18:00.760 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.758+0000 7fd8c82c0640 1 -- 172.21.15.122:0/4194859386 >> v1:172.21.15.204:6800/676228005 conn(0x7fd8a406de00 legacy=0x7fd8a40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:00.760 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.758+0000 7fd8c82c0640 10 monclient: shutdown 2024-03-25T18:18:00.760 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.758+0000 7fd8c82c0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:00.760 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.758+0000 7fd8c82c0640 1 -- 172.21.15.122:0/4194859386 >> v1:172.21.15.122:6789/0 conn(0x7fd8c01430c0 legacy=0x7fd8c01e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:00.761 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.758+0000 7fd8c82c0640 1 -- 172.21.15.122:0/4194859386 shutdown_connections 2024-03-25T18:18:00.761 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.758+0000 7fd8c82c0640 1 -- 172.21.15.122:0/4194859386 >> 172.21.15.122:0/4194859386 conn(0x7fd8c00b7ea0 msgr2=0x7fd8c0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:00.761 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.758+0000 7fd8c82c0640 1 -- 172.21.15.122:0/4194859386 shutdown_connections 2024-03-25T18:18:00.761 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.758+0000 7fd8c82c0640 1 -- 172.21.15.122:0/4194859386 wait complete. 2024-03-25T18:18:00.761 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:00.758+0000 7fd8c82c0640 1 librados: shutdown 2024-03-25T18:18:00.770 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:05.567 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:01:59.309 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:05.578 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:01:59.320 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:05.616 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:01:59.358 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:05.636 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:01:59.378 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:05.641 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:01:59.383 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:05.652 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:01:59.398 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:05.659 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:01:59.404 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:05.687 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:01:59.433 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:05.690 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:01:59.433 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:07.576 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:07.692 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.690+0000 7f4e0fbcf640 10 monclient: get_monmap_and_config 2024-03-25T18:18:07.692 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.690+0000 7f4e0fbcf640 10 monclient: build_initial_monmap 2024-03-25T18:18:07.692 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.690+0000 7f4e0fbcf640 10 monclient: monmap: 2024-03-25T18:18:07.692 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:18:07.693 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:07.693 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:18:07.692019+0000 2024-03-25T18:18:07.693 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:18:07.692019+0000 2024-03-25T18:18:07.693 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:07.693 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:07.693 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:18:07.693 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:18:07.693 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:18:07.693 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:07.693 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.691+0000 7f4e0fbcf640 1 Processor -- start 2024-03-25T18:18:07.693 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.691+0000 7f4e0fbcf640 1 -- start start 2024-03-25T18:18:07.693 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.691+0000 7f4e0fbcf640 10 monclient: init 2024-03-25T18:18:07.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.692+0000 7f4e0fbcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:07.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.692+0000 7f4e0fbcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:07.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.692+0000 7f4e0fbcf640 10 monclient(hunting): picked mon.noname-a con 0x7f4e080a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:18:07.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.692+0000 7f4e0fbcf640 10 monclient(hunting): picked mon.noname-c con 0x7f4e0813f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:18:07.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.692+0000 7f4e0fbcf640 10 monclient(hunting): picked mon.noname-b con 0x7f4e081430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:18:07.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.692+0000 7f4e0fbcf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e08142520 con 0x7f4e080a98f0 2024-03-25T18:18:07.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.692+0000 7f4e0fbcf640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e080b28a0 con 0x7f4e0813f710 2024-03-25T18:18:07.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.692+0000 7f4e0fbcf640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e080b2a30 con 0x7f4e081430c0 2024-03-25T18:18:07.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.692+0000 7f4e0fbcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:07.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.692+0000 7f4e0fbcf640 10 monclient(hunting): authenticate will time out at 7410.958884s 2024-03-25T18:18:07.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.700+0000 7f4e0e16c640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f4e081430c0 0x7f4e0814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:48774/0 (socket says 172.21.15.122:48774) 2024-03-25T18:18:07.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.700+0000 7f4e0e16c640 1 -- 172.21.15.122:0/2476096777 learned_addr learned my addr 172.21.15.122:0/2476096777 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:18:07.711 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.710+0000 7f4e0c969640 1 -- 172.21.15.122:0/2476096777 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2987977440 0 0) 0x7f4e080b2a30 con 0x7f4e081430c0 2024-03-25T18:18:07.711 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.710+0000 7f4e0c969640 10 monclient(hunting): _negotiate 2024-03-25T18:18:07.711 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.710+0000 7f4e0c969640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:07.711 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.710+0000 7f4e0c969640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:07.711 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.710+0000 7f4e0c969640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:18:07.711 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.710+0000 7f4e0c969640 1 -- 172.21.15.122:0/2476096777 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df4003520 con 0x7f4e081430c0 2024-03-25T18:18:07.712 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.711+0000 7f4e0c969640 1 -- 172.21.15.122:0/2476096777 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3020397246 0 0) 0x7f4e080b28a0 con 0x7f4e0813f710 2024-03-25T18:18:07.712 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.711+0000 7f4e0c969640 10 monclient(hunting): _negotiate 2024-03-25T18:18:07.712 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.711+0000 7f4e0c969640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:07.712 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.711+0000 7f4e0c969640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:07.712 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.711+0000 7f4e0c969640 10 monclient(hunting): my global_id is 4297 2024-03-25T18:18:07.712 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.711+0000 7f4e0c969640 1 -- 172.21.15.122:0/2476096777 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e080b2a30 con 0x7f4e0813f710 2024-03-25T18:18:07.722 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.721+0000 7f4e0c969640 1 -- 172.21.15.122:0/2476096777 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3692277153 0 0) 0x7f4e08142520 con 0x7f4e080a98f0 2024-03-25T18:18:07.722 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.721+0000 7f4e0c969640 10 monclient(hunting): _negotiate 2024-03-25T18:18:07.722 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.721+0000 7f4e0c969640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:07.722 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.721+0000 7f4e0c969640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:07.722 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.721+0000 7f4e0c969640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:18:07.722 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.721+0000 7f4e0c969640 1 -- 172.21.15.122:0/2476096777 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e080b28a0 con 0x7f4e080a98f0 2024-03-25T18:18:07.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0c969640 1 -- 172.21.15.122:0/2476096777 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2525966638 0 0) 0x7f4df4003520 con 0x7f4e081430c0 2024-03-25T18:18:07.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0c969640 1 -- 172.21.15.122:0/2476096777 >> v1:172.21.15.204:6789/0 conn(0x7f4e0813f710 legacy=0x7f4e081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0c969640 1 -- 172.21.15.122:0/2476096777 >> v1:172.21.15.122:6789/0 conn(0x7f4e080a98f0 legacy=0x7f4e080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0c969640 10 monclient: _finish_hunting 0 2024-03-25T18:18:07.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0c969640 1 monclient: found mon.noname-b 2024-03-25T18:18:07.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0c969640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:07.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0c969640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:18:07.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0c969640 1 -- 172.21.15.122:0/2476096777 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e080b44c0 con 0x7f4e081430c0 2024-03-25T18:18:07.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0c969640 10 monclient: _finish_auth 0 2024-03-25T18:18:07.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0c969640 10 monclient: _check_auth_tickets 2024-03-25T18:18:07.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0c969640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:07.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0c969640 1 -- 172.21.15.122:0/2476096777 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4e00001a10 con 0x7f4e081430c0 2024-03-25T18:18:07.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0fbcf640 5 monclient: authenticate success, global_id 4280 2024-03-25T18:18:07.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0fbcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:07.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0c969640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:07.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0c969640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:07.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0c969640 10 monclient: dump: 2024-03-25T18:18:07.725 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:07.725 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:07.725 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:07.725 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:07.725 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:07.725 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:07.725 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:07.725 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:07.725 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:07.726 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:07.726 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.722+0000 7f4e0fbcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:07.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.750+0000 7f4e0c969640 1 -- 172.21.15.122:0/2476096777 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e00001b70 con 0x7f4e081430c0 2024-03-25T18:18:07.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.750+0000 7f4e0c969640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:07.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.750+0000 7f4e0c969640 1 -- 172.21.15.122:0/2476096777 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4e00001f80 con 0x7f4e081430c0 2024-03-25T18:18:07.752 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.750+0000 7f4e0c969640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:07.752 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.750+0000 7f4e0c969640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:07.752 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.750+0000 7f4e0c969640 10 monclient: dump: 2024-03-25T18:18:07.752 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:07.752 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:07.752 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:07.752 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:07.752 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:07.752 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:07.752 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:07.752 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:07.752 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:07.752 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:07.752 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.750+0000 7f4e0fbcf640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:07.753 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.750+0000 7f4e0fbcf640 4 set_mon_vals no callback set 2024-03-25T18:18:07.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.751+0000 7f4e0fbcf640 10 monclient: shutdown 2024-03-25T18:18:07.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.751+0000 7f4e0fbcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:07.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.751+0000 7f4e0fbcf640 1 -- 172.21.15.122:0/2476096777 >> v1:172.21.15.122:6790/0 conn(0x7f4e081430c0 legacy=0x7f4e0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.751+0000 7f4e0fbcf640 1 -- 172.21.15.122:0/2476096777 shutdown_connections 2024-03-25T18:18:07.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.751+0000 7f4e0fbcf640 1 -- 172.21.15.122:0/2476096777 >> 172.21.15.122:0/2476096777 conn(0x7f4e080b7ea0 msgr2=0x7f4e080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:07.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.751+0000 7f4e0fbcf640 1 -- 172.21.15.122:0/2476096777 shutdown_connections 2024-03-25T18:18:07.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.751+0000 7f4e0fbcf640 1 -- 172.21.15.122:0/2476096777 wait complete. 2024-03-25T18:18:07.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.752+0000 7f4e0fbcf640 10 monclient: build_initial_monmap 2024-03-25T18:18:07.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.752+0000 7f4e0fbcf640 10 monclient: monmap: 2024-03-25T18:18:07.755 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:18:07.756 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:07.756 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:18:07.756 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:18:07.756 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:07.756 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:18:07.756 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:18:07.756 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:18:07.756 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:18:07.756 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:07.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.752+0000 7f4e0fbcf640 1 librados: starting msgr at 2024-03-25T18:18:07.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.752+0000 7f4e0fbcf640 1 librados: starting objecter 2024-03-25T18:18:07.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.753+0000 7f4e0fbcf640 1 Processor -- start 2024-03-25T18:18:07.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.753+0000 7f4e0fbcf640 1 -- start start 2024-03-25T18:18:07.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.753+0000 7f4e0fbcf640 1 librados: setting wanted keys 2024-03-25T18:18:07.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.753+0000 7f4e0fbcf640 1 librados: calling monclient init 2024-03-25T18:18:07.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.753+0000 7f4e0fbcf640 10 monclient: init 2024-03-25T18:18:07.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.753+0000 7f4e0fbcf640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:07.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.753+0000 7f4e0fbcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:07.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.753+0000 7f4e0fbcf640 10 monclient(hunting): picked mon.noname-a con 0x7f4e080a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:18:07.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.753+0000 7f4e0fbcf640 10 monclient(hunting): picked mon.noname-b con 0x7f4e0813f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:18:07.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.753+0000 7f4e0fbcf640 10 monclient(hunting): picked mon.noname-c con 0x7f4e081430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:18:07.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.753+0000 7f4e0fbcf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e081daca0 con 0x7f4e080a98f0 2024-03-25T18:18:07.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.753+0000 7f4e0fbcf640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e08138ab0 con 0x7f4e0813f710 2024-03-25T18:18:07.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.753+0000 7f4e0fbcf640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e080aba90 con 0x7f4e081430c0 2024-03-25T18:18:07.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.753+0000 7f4e0fbcf640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:07.757 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.753+0000 7f4e0fbcf640 10 monclient(hunting): authenticate will time out at 7411.019938s 2024-03-25T18:18:07.759 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.758+0000 7f4e0d16a640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f4e0813f710 0x7f4e081db130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:57820/0 (socket says 172.21.15.122:57820) 2024-03-25T18:18:07.760 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.759+0000 7f4e0d16a640 1 -- 172.21.15.122:0/3611919506 learned_addr learned my addr 172.21.15.122:0/3611919506 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:18:07.761 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.760+0000 7f4dfe7fc640 10 client.?.objecter ms_handle_connect 0x7f4e0813f710 2024-03-25T18:18:07.761 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.760+0000 7f4dfe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:07.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.765+0000 7f4dfe7fc640 10 client.?.objecter ms_handle_connect 0x7f4e081430c0 2024-03-25T18:18:07.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.765+0000 7f4dfe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:07.768 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.767+0000 7f4dfe7fc640 10 client.?.objecter ms_handle_connect 0x7f4e080a98f0 2024-03-25T18:18:07.768 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.767+0000 7f4dfe7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:07.772 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.771+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3074203469 0 0) 0x7f4e080aba90 con 0x7f4e081430c0 2024-03-25T18:18:07.772 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.771+0000 7f4dfe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:07.772 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.771+0000 7f4dfe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:07.772 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.771+0000 7f4dfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:07.772 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.771+0000 7f4dfe7fc640 10 monclient(hunting): my global_id is 4283 2024-03-25T18:18:07.772 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.771+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dec004eb0 con 0x7f4e081430c0 2024-03-25T18:18:07.780 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.779+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281739178 0 0) 0x7f4e08138ab0 con 0x7f4e0813f710 2024-03-25T18:18:07.780 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.779+0000 7f4dfe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:07.780 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.779+0000 7f4dfe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:07.781 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.779+0000 7f4dfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:07.781 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.779+0000 7f4dfe7fc640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:18:07.781 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.779+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e080aba90 con 0x7f4e0813f710 2024-03-25T18:18:07.785 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.784+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284729688 0 0) 0x7f4e081daca0 con 0x7f4e080a98f0 2024-03-25T18:18:07.785 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.784+0000 7f4dfe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:07.785 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.784+0000 7f4dfe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:07.785 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.784+0000 7f4dfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:07.785 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.784+0000 7f4dfe7fc640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:18:07.785 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.784+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e08138ab0 con 0x7f4e080a98f0 2024-03-25T18:18:07.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.787+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2099670383 0 0) 0x7f4e080aba90 con 0x7f4e0813f710 2024-03-25T18:18:07.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.787+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e081daca0 con 0x7f4e0813f710 2024-03-25T18:18:07.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.787+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4df80019b0 con 0x7f4e0813f710 2024-03-25T18:18:07.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.787+0000 7f4dfe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:07.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.787+0000 7f4dfe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:07.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.787+0000 7f4dfe7fc640 10 monclient(hunting): dump: 2024-03-25T18:18:07.789 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:07.789 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:07.789 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:07.789 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:07.789 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:07.789 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:07.790 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:07.790 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:07.790 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:07.790 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:07.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.787+0000 7f4dfe7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:07.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.789+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3416309930 0 0) 0x7f4dec004eb0 con 0x7f4e081430c0 2024-03-25T18:18:07.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.789+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e080aba90 con 0x7f4e081430c0 2024-03-25T18:18:07.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.789+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4e00001e80 con 0x7f4e081430c0 2024-03-25T18:18:07.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.789+0000 7f4dfe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:07.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.789+0000 7f4dfe7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:07.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.789+0000 7f4dfe7fc640 10 monclient(hunting): dump: 2024-03-25T18:18:07.791 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:07.791 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:07.791 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:07.791 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:07.791 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:07.791 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:07.791 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:07.791 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:07.791 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:07.791 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:07.799 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.797+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2686028140 0 0) 0x7f4e081daca0 con 0x7f4e0813f710 2024-03-25T18:18:07.799 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.797+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 >> v1:172.21.15.122:6790/0 conn(0x7f4e081430c0 legacy=0x7f4e081d47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.799 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.797+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 >> v1:172.21.15.122:6789/0 conn(0x7f4e080a98f0 legacy=0x7f4e081383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.799 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.797+0000 7f4dfe7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:07.799 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.797+0000 7f4dfe7fc640 1 monclient: found mon.b 2024-03-25T18:18:07.799 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.797+0000 7f4dfe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:07.799 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.797+0000 7f4dfe7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:07.799 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.797+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e081d41e0 con 0x7f4e0813f710 2024-03-25T18:18:07.799 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.797+0000 7f4dfe7fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:07.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.797+0000 7f4dfe7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:07.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.797+0000 7f4dfe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:07.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.797+0000 7f4e0fbcf640 5 monclient: authenticate success, global_id 4300 2024-03-25T18:18:07.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.798+0000 7f4e0fbcf640 10 monclient: _renew_subs 2024-03-25T18:18:07.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.798+0000 7f4e0fbcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:07.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.798+0000 7f4e0fbcf640 1 -- 172.21.15.122:0/3611919506 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e081db840 con 0x7f4e0813f710 2024-03-25T18:18:07.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.798+0000 7f4e0fbcf640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:07.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.798+0000 7f4e0fbcf640 10 monclient: _renew_subs 2024-03-25T18:18:07.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.798+0000 7f4e0fbcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:07.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.798+0000 7f4e0fbcf640 1 -- 172.21.15.122:0/3611919506 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e08138f00 con 0x7f4e0813f710 2024-03-25T18:18:07.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.798+0000 7f4e0fbcf640 1 librados: init done 2024-03-25T18:18:07.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.799+0000 7f4e0fbcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:07.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.799+0000 7f4e0fbcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:07.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.799+0000 7f4e0fbcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:07.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:07.799+0000 7f4e0fbcf640 1 -- 172.21.15.122:0/3611919506 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e081dae50 con 0x7f4e0813f710 2024-03-25T18:18:08.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.085+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4df8003610 con 0x7f4e0813f710 2024-03-25T18:18:08.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.085+0000 7f4dfe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:08.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.085+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4df8003800 con 0x7f4e0813f710 2024-03-25T18:18:08.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.085+0000 7f4dfe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:08.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.085+0000 7f4dfe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:08.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.085+0000 7f4dfe7fc640 10 monclient: dump: 2024-03-25T18:18:08.087 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:08.087 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:08.087 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:08.087 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:08.088 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:08.088 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:08.088 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:08.088 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:08.088 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:08.088 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:08.088 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.085+0000 7f4dfffff640 4 set_mon_vals no callback set 2024-03-25T18:18:08.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.086+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f4df8003a20 con 0x7f4e0813f710 2024-03-25T18:18:08.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.087+0000 7f4e0d96b640 10 monclient: get_auth_request con 0x7f4dec06df70 auth_method 0 2024-03-25T18:18:08.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.087+0000 7f4dfe7fc640 10 client.4300.objecter ms_handle_connect 0x7f4dec06df70 2024-03-25T18:18:08.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.092+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f4df8054670 con 0x7f4e0813f710 2024-03-25T18:18:08.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.092+0000 7f4dfe7fc640 10 client.4300.objecter ms_dispatch 0x7f4e08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:08.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.092+0000 7f4dfe7fc640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:08.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.092+0000 7f4dfe7fc640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:08.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.092+0000 7f4dfe7fc640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-25T18:18:08.094 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.093+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4df80b18b0 con 0x7f4e0813f710 2024-03-25T18:18:08.094 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.093+0000 7f4dfe7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:08.094 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.093+0000 7f4dfe7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:08.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.352+0000 7f4e0fbcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:08.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.352+0000 7f4e0fbcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:08.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.352+0000 7f4e0fbcf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:08.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.352+0000 7f4e0fbcf640 1 -- 172.21.15.122:0/3611919506 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e080627b0 con 0x7f4e0813f710 2024-03-25T18:18:08.373 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.372+0000 7f4dfe7fc640 1 -- 172.21.15.122:0/3611919506 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f4df8050180 con 0x7f4e0813f710 2024-03-25T18:18:08.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.372+0000 7f4dfe7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:08.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.372+0000 7f4dfe7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:08.374 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:18:08.374 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:08.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.375+0000 7f4e0fbcf640 10 librados: watch_flush enter 2024-03-25T18:18:08.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.375+0000 7f4e0fbcf640 10 librados: watch_flush exit 2024-03-25T18:18:08.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.375+0000 7f4e0fbcf640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-25T18:18:08.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.375+0000 7f4e0fbcf640 10 client.4300.objecter successfully canceled tick 2024-03-25T18:18:08.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.375+0000 7f4e0fbcf640 1 -- 172.21.15.122:0/3611919506 >> v1:172.21.15.204:6800/676228005 conn(0x7f4dec06df70 legacy=0x7f4dec070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:08.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.375+0000 7f4e0fbcf640 10 monclient: shutdown 2024-03-25T18:18:08.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.375+0000 7f4e0fbcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:08.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.375+0000 7f4e0fbcf640 1 -- 172.21.15.122:0/3611919506 >> v1:172.21.15.204:6789/0 conn(0x7f4e0813f710 legacy=0x7f4e081db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:08.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.375+0000 7f4e0fbcf640 1 -- 172.21.15.122:0/3611919506 shutdown_connections 2024-03-25T18:18:08.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.375+0000 7f4e0fbcf640 1 -- 172.21.15.122:0/3611919506 >> 172.21.15.122:0/3611919506 conn(0x7f4e080b7ea0 msgr2=0x7f4e08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:08.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.375+0000 7f4e0fbcf640 1 -- 172.21.15.122:0/3611919506 shutdown_connections 2024-03-25T18:18:08.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.375+0000 7f4e0fbcf640 1 -- 172.21.15.122:0/3611919506 wait complete. 2024-03-25T18:18:08.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:08.375+0000 7f4e0fbcf640 1 librados: shutdown 2024-03-25T18:18:08.387 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:15.194 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:15.309 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.308+0000 7f5699ed9640 10 monclient: get_monmap_and_config 2024-03-25T18:18:15.309 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.308+0000 7f5699ed9640 10 monclient: build_initial_monmap 2024-03-25T18:18:15.309 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.308+0000 7f5699ed9640 10 monclient: monmap: 2024-03-25T18:18:15.309 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:18:15.309 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:15.309 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:18:15.309756+0000 2024-03-25T18:18:15.310 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:18:15.309756+0000 2024-03-25T18:18:15.310 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:15.310 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:15.310 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:18:15.310 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:18:15.310 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:18:15.310 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:15.311 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.309+0000 7f5699ed9640 1 Processor -- start 2024-03-25T18:18:15.311 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.309+0000 7f5699ed9640 1 -- start start 2024-03-25T18:18:15.311 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.309+0000 7f5699ed9640 10 monclient: init 2024-03-25T18:18:15.311 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.310+0000 7f5699ed9640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:15.311 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.310+0000 7f5699ed9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:15.311 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.310+0000 7f5699ed9640 10 monclient(hunting): picked mon.noname-b con 0x7f56940b4d60 addr v1:172.21.15.122:6790/0 2024-03-25T18:18:15.311 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.310+0000 7f5699ed9640 10 monclient(hunting): picked mon.noname-a con 0x7f56940b1730 addr v1:172.21.15.122:6789/0 2024-03-25T18:18:15.311 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.310+0000 7f5699ed9640 10 monclient(hunting): picked mon.noname-c con 0x7f569414be90 addr v1:172.21.15.204:6789/0 2024-03-25T18:18:15.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.310+0000 7f5699ed9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56940af940 con 0x7f56940b1730 2024-03-25T18:18:15.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.310+0000 7f5699ed9640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56940b0a70 con 0x7f569414be90 2024-03-25T18:18:15.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.310+0000 7f5699ed9640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56940b0d50 con 0x7f56940b4d60 2024-03-25T18:18:15.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.310+0000 7f5699ed9640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:15.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.310+0000 7f5699ed9640 10 monclient(hunting): authenticate will time out at 7418.576338s 2024-03-25T18:18:15.319 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.318+0000 7f5693fff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f569414be90 0x7f569414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:41806/0 (socket says 172.21.15.122:41806) 2024-03-25T18:18:15.319 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.318+0000 7f5693fff640 1 -- 172.21.15.122:0/2039264599 learned_addr learned my addr 172.21.15.122:0/2039264599 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:18:15.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.333+0000 7f56927fc640 1 -- 172.21.15.122:0/2039264599 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357491257 0 0) 0x7f56940b0a70 con 0x7f569414be90 2024-03-25T18:18:15.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.333+0000 7f56927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:15.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.333+0000 7f56927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:15.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.333+0000 7f56927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:15.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.333+0000 7f56927fc640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:18:15.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.333+0000 7f56927fc640 1 -- 172.21.15.122:0/2039264599 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f567c003520 con 0x7f569414be90 2024-03-25T18:18:15.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.343+0000 7f56927fc640 1 -- 172.21.15.122:0/2039264599 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149693373 0 0) 0x7f56940af940 con 0x7f56940b1730 2024-03-25T18:18:15.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.343+0000 7f56927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:15.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.343+0000 7f56927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:15.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.343+0000 7f56927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:15.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.343+0000 7f56927fc640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:18:15.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.343+0000 7f56927fc640 1 -- 172.21.15.122:0/2039264599 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56940b0a70 con 0x7f56940b1730 2024-03-25T18:18:15.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.343+0000 7f56927fc640 1 -- 172.21.15.122:0/2039264599 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138192128 0 0) 0x7f56940b0d50 con 0x7f56940b4d60 2024-03-25T18:18:15.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.343+0000 7f56927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:15.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.343+0000 7f56927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:15.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.343+0000 7f56927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:15.345 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.343+0000 7f56927fc640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:18:15.345 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.343+0000 7f56927fc640 1 -- 172.21.15.122:0/2039264599 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56940af940 con 0x7f56940b4d60 2024-03-25T18:18:15.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.346+0000 7f56927fc640 1 -- 172.21.15.122:0/2039264599 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3745903172 0 0) 0x7f567c003520 con 0x7f569414be90 2024-03-25T18:18:15.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.346+0000 7f56927fc640 1 -- 172.21.15.122:0/2039264599 >> v1:172.21.15.122:6790/0 conn(0x7f56940b4d60 legacy=0x7f569413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.346+0000 7f56927fc640 1 -- 172.21.15.122:0/2039264599 >> v1:172.21.15.122:6789/0 conn(0x7f56940b1730 legacy=0x7f56940b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.346+0000 7f56927fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:15.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.346+0000 7f56927fc640 1 monclient: found mon.noname-c 2024-03-25T18:18:15.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.346+0000 7f56927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:15.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.346+0000 7f56927fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:18:15.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.346+0000 7f56927fc640 1 -- 172.21.15.122:0/2039264599 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56940b1030 con 0x7f569414be90 2024-03-25T18:18:15.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.346+0000 7f56927fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:15.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.346+0000 7f56927fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:15.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.346+0000 7f56927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:15.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.346+0000 7f56927fc640 1 -- 172.21.15.122:0/2039264599 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f568c001a10 con 0x7f569414be90 2024-03-25T18:18:15.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.346+0000 7f56927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:15.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.346+0000 7f56927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:15.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.346+0000 7f56927fc640 10 monclient: dump: 2024-03-25T18:18:15.349 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:15.349 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:15.349 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:15.349 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:15.349 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:15.349 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:15.349 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:15.349 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:15.349 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:15.349 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:15.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.346+0000 7f5699ed9640 5 monclient: authenticate success, global_id 4303 2024-03-25T18:18:15.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.346+0000 7f5699ed9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:15.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.352+0000 7f56927fc640 1 -- 172.21.15.122:0/2039264599 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f568c001c30 con 0x7f569414be90 2024-03-25T18:18:15.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.352+0000 7f56927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:15.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.352+0000 7f56927fc640 1 -- 172.21.15.122:0/2039264599 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f568c001fc0 con 0x7f569414be90 2024-03-25T18:18:15.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.352+0000 7f56927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:15.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.352+0000 7f56927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:15.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.352+0000 7f56927fc640 10 monclient: dump: 2024-03-25T18:18:15.353 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:15.353 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:15.353 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:15.353 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:15.353 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:15.354 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:15.354 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:15.354 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:15.354 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:15.354 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:15.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.352+0000 7f5699ed9640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:15.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.352+0000 7f5699ed9640 4 set_mon_vals no callback set 2024-03-25T18:18:15.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.353+0000 7f5699ed9640 10 monclient: shutdown 2024-03-25T18:18:15.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.353+0000 7f5699ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:15.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.353+0000 7f5699ed9640 1 -- 172.21.15.122:0/2039264599 >> v1:172.21.15.204:6789/0 conn(0x7f569414be90 legacy=0x7f569414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.353+0000 7f5699ed9640 1 -- 172.21.15.122:0/2039264599 shutdown_connections 2024-03-25T18:18:15.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.353+0000 7f5699ed9640 1 -- 172.21.15.122:0/2039264599 >> 172.21.15.122:0/2039264599 conn(0x7f56940617a0 msgr2=0x7f5694061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:15.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.353+0000 7f5699ed9640 1 -- 172.21.15.122:0/2039264599 shutdown_connections 2024-03-25T18:18:15.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.353+0000 7f5699ed9640 1 -- 172.21.15.122:0/2039264599 wait complete. 2024-03-25T18:18:15.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.353+0000 7f5699ed9640 10 monclient: build_initial_monmap 2024-03-25T18:18:15.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.353+0000 7f5699ed9640 10 monclient: monmap: 2024-03-25T18:18:15.356 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:18:15.356 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:15.356 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:18:15.356 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:18:15.356 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:15.356 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:18:15.356 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:18:15.356 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:18:15.356 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:18:15.356 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:15.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.354+0000 7f5699ed9640 1 librados: starting msgr at 2024-03-25T18:18:15.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.354+0000 7f5699ed9640 1 librados: starting objecter 2024-03-25T18:18:15.357 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.354+0000 7f5699ed9640 1 Processor -- start 2024-03-25T18:18:15.357 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.354+0000 7f5699ed9640 1 -- start start 2024-03-25T18:18:15.357 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.354+0000 7f5699ed9640 1 librados: setting wanted keys 2024-03-25T18:18:15.357 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.354+0000 7f5699ed9640 1 librados: calling monclient init 2024-03-25T18:18:15.357 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.354+0000 7f5699ed9640 10 monclient: init 2024-03-25T18:18:15.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.355+0000 7f5699ed9640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:15.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.355+0000 7f5699ed9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:15.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.355+0000 7f5699ed9640 10 monclient(hunting): picked mon.noname-c con 0x7f56940b1730 addr v1:172.21.15.122:6790/0 2024-03-25T18:18:15.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.355+0000 7f5699ed9640 10 monclient(hunting): picked mon.noname-b con 0x7f56940b4d60 addr v1:172.21.15.204:6789/0 2024-03-25T18:18:15.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.355+0000 7f5699ed9640 10 monclient(hunting): picked mon.noname-a con 0x7f569414be90 addr v1:172.21.15.122:6789/0 2024-03-25T18:18:15.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.355+0000 7f5699ed9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5694140380 con 0x7f569414be90 2024-03-25T18:18:15.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.355+0000 7f5699ed9640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5694140660 con 0x7f56940b4d60 2024-03-25T18:18:15.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.355+0000 7f5699ed9640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5694141e30 con 0x7f56940b1730 2024-03-25T18:18:15.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.355+0000 7f5699ed9640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:15.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.355+0000 7f5699ed9640 10 monclient(hunting): authenticate will time out at 7418.621261s 2024-03-25T18:18:15.369 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.368+0000 7f56937fe640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f56940b1730 0x7f5694138510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:46838/0 (socket says 172.21.15.122:46838) 2024-03-25T18:18:15.369 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.368+0000 7f56937fe640 1 -- 172.21.15.122:0/931992204 learned_addr learned my addr 172.21.15.122:0/931992204 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:18:15.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.371+0000 7f567bfff640 10 client.?.objecter ms_handle_connect 0x7f56940b4d60 2024-03-25T18:18:15.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.371+0000 7f567bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:15.373 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.372+0000 7f567bfff640 10 client.?.objecter ms_handle_connect 0x7f569414be90 2024-03-25T18:18:15.373 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.372+0000 7f567bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:15.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.373+0000 7f567bfff640 10 client.?.objecter ms_handle_connect 0x7f56940b1730 2024-03-25T18:18:15.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.373+0000 7f567bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:15.379 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.377+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127623406 0 0) 0x7f5694140660 con 0x7f56940b4d60 2024-03-25T18:18:15.379 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.377+0000 7f567bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:15.379 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.377+0000 7f567bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:15.379 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.377+0000 7f567bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:15.379 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.377+0000 7f567bfff640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:18:15.379 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.377+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5680004eb0 con 0x7f56940b4d60 2024-03-25T18:18:15.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.379+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4151930708 0 0) 0x7f5694140380 con 0x7f569414be90 2024-03-25T18:18:15.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.379+0000 7f567bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:15.381 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.379+0000 7f567bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:15.381 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.379+0000 7f567bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:15.381 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.379+0000 7f567bfff640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:18:15.381 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.379+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5694140660 con 0x7f569414be90 2024-03-25T18:18:15.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.388+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2646071055 0 0) 0x7f5694141e30 con 0x7f56940b1730 2024-03-25T18:18:15.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.388+0000 7f567bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:15.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.388+0000 7f567bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:15.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.388+0000 7f567bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:15.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.388+0000 7f567bfff640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:18:15.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.388+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5694140380 con 0x7f56940b1730 2024-03-25T18:18:15.393 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.392+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 892316146 0 0) 0x7f5680004eb0 con 0x7f56940b4d60 2024-03-25T18:18:15.393 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.392+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5694141e30 con 0x7f56940b4d60 2024-03-25T18:18:15.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.392+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5688001970 con 0x7f56940b4d60 2024-03-25T18:18:15.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.392+0000 7f567bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:15.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.392+0000 7f567bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:15.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.392+0000 7f567bfff640 10 monclient(hunting): dump: 2024-03-25T18:18:15.394 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:15.394 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:15.394 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:15.394 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:15.394 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:15.394 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:15.394 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:15.394 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:15.394 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:15.394 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:15.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.392+0000 7f567bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:15.402 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1730200086 0 0) 0x7f5694141e30 con 0x7f56940b4d60 2024-03-25T18:18:15.402 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 >> v1:172.21.15.122:6790/0 conn(0x7f56940b1730 legacy=0x7f5694138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.402 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 >> v1:172.21.15.122:6789/0 conn(0x7f569414be90 legacy=0x7f56941e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.402 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f567bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:15.402 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f567bfff640 1 monclient: found mon.b 2024-03-25T18:18:15.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f567bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:15.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f567bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:15.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5694142110 con 0x7f56940b4d60 2024-03-25T18:18:15.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f567bfff640 10 monclient: _finish_auth 0 2024-03-25T18:18:15.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f567bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:15.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f567bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:15.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f5699ed9640 5 monclient: authenticate success, global_id 4306 2024-03-25T18:18:15.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f5699ed9640 10 monclient: _renew_subs 2024-03-25T18:18:15.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f5699ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:15.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f5699ed9640 1 -- 172.21.15.122:0/931992204 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5694140e60 con 0x7f56940b4d60 2024-03-25T18:18:15.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f5699ed9640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:15.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f5699ed9640 10 monclient: _renew_subs 2024-03-25T18:18:15.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f5699ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:15.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f5699ed9640 1 -- 172.21.15.122:0/931992204 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5694142c40 con 0x7f56940b4d60 2024-03-25T18:18:15.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.401+0000 7f5699ed9640 1 librados: init done 2024-03-25T18:18:15.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.402+0000 7f5699ed9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:15.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.402+0000 7f5699ed9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:15.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.402+0000 7f5699ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:15.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.402+0000 7f5699ed9640 1 -- 172.21.15.122:0/931992204 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56940627b0 con 0x7f56940b4d60 2024-03-25T18:18:15.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.669+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5688003610 con 0x7f56940b4d60 2024-03-25T18:18:15.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.669+0000 7f567bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:15.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.669+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5688003920 con 0x7f56940b4d60 2024-03-25T18:18:15.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.669+0000 7f567bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:15.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.669+0000 7f567bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:15.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.669+0000 7f567bfff640 10 monclient: dump: 2024-03-25T18:18:15.670 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:15.671 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:15.673 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:15.673 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:15.673 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:15.673 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:15.673 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:15.673 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:15.673 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:15.673 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:15.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.669+0000 7f5691ffb640 4 set_mon_vals no callback set 2024-03-25T18:18:15.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.670+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f5688019930 con 0x7f56940b4d60 2024-03-25T18:18:15.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.670+0000 7f56937fe640 10 monclient: get_auth_request con 0x7f568006dda0 auth_method 0 2024-03-25T18:18:15.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.670+0000 7f567bfff640 10 client.4306.objecter ms_handle_connect 0x7f568006dda0 2024-03-25T18:18:15.675 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.674+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f5688053660 con 0x7f56940b4d60 2024-03-25T18:18:15.675 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.674+0000 7f567bfff640 10 client.4306.objecter ms_dispatch 0x7f5694000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:15.675 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.674+0000 7f567bfff640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:15.675 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.674+0000 7f567bfff640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:15.675 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.674+0000 7f567bfff640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-25T18:18:15.677 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.676+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f568804efa0 con 0x7f56940b4d60 2024-03-25T18:18:15.677 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.676+0000 7f567bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:15.677 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.676+0000 7f567bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:15.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.931+0000 7f5699ed9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:15.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.931+0000 7f5699ed9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:15.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.931+0000 7f5699ed9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:15.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.931+0000 7f5699ed9640 1 -- 172.21.15.122:0/931992204 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5694140810 con 0x7f56940b4d60 2024-03-25T18:18:15.966 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.964+0000 7f567bfff640 1 -- 172.21.15.122:0/931992204 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f5694140810 con 0x7f56940b4d60 2024-03-25T18:18:15.966 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.965+0000 7f567bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:15.966 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.965+0000 7f567bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:15.966 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:18:15.966 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:15.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.967+0000 7f5699ed9640 10 librados: watch_flush enter 2024-03-25T18:18:15.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.967+0000 7f5699ed9640 10 librados: watch_flush exit 2024-03-25T18:18:15.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.967+0000 7f5699ed9640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-25T18:18:15.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.967+0000 7f5699ed9640 10 client.4306.objecter successfully canceled tick 2024-03-25T18:18:15.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.967+0000 7f5699ed9640 1 -- 172.21.15.122:0/931992204 >> v1:172.21.15.204:6800/676228005 conn(0x7f568006dda0 legacy=0x7f5680070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.967+0000 7f5699ed9640 10 monclient: shutdown 2024-03-25T18:18:15.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.967+0000 7f5699ed9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:15.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.967+0000 7f5699ed9640 1 -- 172.21.15.122:0/931992204 >> v1:172.21.15.204:6789/0 conn(0x7f56940b4d60 legacy=0x7f5694142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.967+0000 7f5699ed9640 1 -- 172.21.15.122:0/931992204 shutdown_connections 2024-03-25T18:18:15.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.967+0000 7f5699ed9640 1 -- 172.21.15.122:0/931992204 >> 172.21.15.122:0/931992204 conn(0x7f56940617a0 msgr2=0x7f56940b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:15.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.967+0000 7f5699ed9640 1 -- 172.21.15.122:0/931992204 shutdown_connections 2024-03-25T18:18:15.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.967+0000 7f5699ed9640 1 -- 172.21.15.122:0/931992204 wait complete. 2024-03-25T18:18:15.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:15.967+0000 7f5699ed9640 1 librados: shutdown 2024-03-25T18:18:15.981 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:22.787 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:22.885 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.884+0000 7f9ff00f6640 10 monclient: get_monmap_and_config 2024-03-25T18:18:22.885 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.884+0000 7f9ff00f6640 10 monclient: build_initial_monmap 2024-03-25T18:18:22.885 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.884+0000 7f9ff00f6640 10 monclient: monmap: 2024-03-25T18:18:22.885 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:18:22.885 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:22.885 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:18:22.885389+0000 2024-03-25T18:18:22.885 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:18:22.885389+0000 2024-03-25T18:18:22.885 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:22.885 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:22.886 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:18:22.886 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:18:22.886 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:18:22.886 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:22.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.884+0000 7f9ff00f6640 1 Processor -- start 2024-03-25T18:18:22.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.885+0000 7f9ff00f6640 1 -- start start 2024-03-25T18:18:22.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.885+0000 7f9ff00f6640 10 monclient: init 2024-03-25T18:18:22.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.885+0000 7f9ff00f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:22.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.885+0000 7f9ff00f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:22.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.885+0000 7f9ff00f6640 10 monclient(hunting): picked mon.noname-c con 0x7f9fe80a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:18:22.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.886+0000 7f9ff00f6640 10 monclient(hunting): picked mon.noname-a con 0x7f9fe813f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:18:22.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.886+0000 7f9ff00f6640 10 monclient(hunting): picked mon.noname-b con 0x7f9fe81430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:18:22.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.886+0000 7f9ff00f6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe8142520 con 0x7f9fe813f710 2024-03-25T18:18:22.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.886+0000 7f9ff00f6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe80b28a0 con 0x7f9fe80a98f0 2024-03-25T18:18:22.888 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.886+0000 7f9ff00f6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe80b2a30 con 0x7f9fe81430c0 2024-03-25T18:18:22.888 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.886+0000 7f9ff00f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:22.888 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.886+0000 7f9ff00f6640 10 monclient(hunting): authenticate will time out at 7426.152304s 2024-03-25T18:18:22.902 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.901+0000 7f9fed691640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9fe813f710 0x7f9fe81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:48368/0 (socket says 172.21.15.122:48368) 2024-03-25T18:18:22.902 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.901+0000 7f9fed691640 1 -- 172.21.15.122:0/2983896027 learned_addr learned my addr 172.21.15.122:0/2983896027 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:18:22.922 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.921+0000 7f9fece90640 1 -- 172.21.15.122:0/2983896027 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656530096 0 0) 0x7f9fe80b28a0 con 0x7f9fe80a98f0 2024-03-25T18:18:22.922 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.921+0000 7f9fece90640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.922 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.921+0000 7f9fece90640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.922 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.921+0000 7f9fece90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.922 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.921+0000 7f9fece90640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:18:22.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.921+0000 7f9fece90640 1 -- 172.21.15.122:0/2983896027 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd4003520 con 0x7f9fe80a98f0 2024-03-25T18:18:22.927 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.926+0000 7f9fece90640 1 -- 172.21.15.122:0/2983896027 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2639109444 0 0) 0x7f9fe80b2a30 con 0x7f9fe81430c0 2024-03-25T18:18:22.927 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.926+0000 7f9fece90640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.927 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.926+0000 7f9fece90640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.927 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.926+0000 7f9fece90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.927 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.926+0000 7f9fece90640 10 monclient(hunting): my global_id is 4292 2024-03-25T18:18:22.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.926+0000 7f9fece90640 1 -- 172.21.15.122:0/2983896027 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe80b28a0 con 0x7f9fe81430c0 2024-03-25T18:18:22.929 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.928+0000 7f9fece90640 1 -- 172.21.15.122:0/2983896027 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2280660180 0 0) 0x7f9fe8142520 con 0x7f9fe813f710 2024-03-25T18:18:22.930 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.928+0000 7f9fece90640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.930 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.928+0000 7f9fece90640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.930 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.928+0000 7f9fece90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.930 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.928+0000 7f9fece90640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:18:22.930 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.928+0000 7f9fece90640 1 -- 172.21.15.122:0/2983896027 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe80b2a30 con 0x7f9fe813f710 2024-03-25T18:18:22.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.930+0000 7f9fece90640 1 -- 172.21.15.122:0/2983896027 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3844886259 0 0) 0x7f9fd4003520 con 0x7f9fe80a98f0 2024-03-25T18:18:22.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.930+0000 7f9fece90640 1 -- 172.21.15.122:0/2983896027 >> v1:172.21.15.122:6790/0 conn(0x7f9fe81430c0 legacy=0x7f9fe814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.930+0000 7f9fece90640 1 -- 172.21.15.122:0/2983896027 >> v1:172.21.15.122:6789/0 conn(0x7f9fe813f710 legacy=0x7f9fe81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.930+0000 7f9fece90640 10 monclient: _finish_hunting 0 2024-03-25T18:18:22.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.930+0000 7f9fece90640 1 monclient: found mon.noname-c 2024-03-25T18:18:22.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.930+0000 7f9fece90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:22.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.930+0000 7f9fece90640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:18:22.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.930+0000 7f9fece90640 1 -- 172.21.15.122:0/2983896027 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe80b44c0 con 0x7f9fe80a98f0 2024-03-25T18:18:22.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.930+0000 7f9fece90640 10 monclient: _finish_auth 0 2024-03-25T18:18:22.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.930+0000 7f9fece90640 10 monclient: _check_auth_tickets 2024-03-25T18:18:22.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.930+0000 7f9fece90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:22.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.930+0000 7f9fece90640 1 -- 172.21.15.122:0/2983896027 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9fe4001690 con 0x7f9fe80a98f0 2024-03-25T18:18:22.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.930+0000 7f9fece90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:22.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.930+0000 7f9fece90640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:22.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.930+0000 7f9fece90640 10 monclient: dump: 2024-03-25T18:18:22.932 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:22.932 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:22.932 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:22.932 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:22.932 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:22.933 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:22.933 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:22.933 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:22.933 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:22.933 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:22.933 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.930+0000 7f9ff00f6640 5 monclient: authenticate success, global_id 4309 2024-03-25T18:18:22.933 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.930+0000 7f9ff00f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:22.937 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.936+0000 7f9fece90640 1 -- 172.21.15.122:0/2983896027 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fe40017f0 con 0x7f9fe80a98f0 2024-03-25T18:18:22.937 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.936+0000 7f9fece90640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:22.937 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.936+0000 7f9fece90640 1 -- 172.21.15.122:0/2983896027 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9fe4001c00 con 0x7f9fe80a98f0 2024-03-25T18:18:22.938 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.936+0000 7f9fece90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:22.938 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.936+0000 7f9fece90640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:22.938 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.936+0000 7f9fece90640 10 monclient: dump: 2024-03-25T18:18:22.938 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:22.938 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:22.938 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:22.938 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:22.938 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:22.938 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:22.938 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:22.938 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:22.938 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:22.938 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:22.938 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.936+0000 7f9ff00f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:22.938 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.936+0000 7f9ff00f6640 4 set_mon_vals no callback set 2024-03-25T18:18:22.940 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.937+0000 7f9ff00f6640 10 monclient: shutdown 2024-03-25T18:18:22.940 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.937+0000 7f9ff00f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:22.940 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.937+0000 7f9ff00f6640 1 -- 172.21.15.122:0/2983896027 >> v1:172.21.15.204:6789/0 conn(0x7f9fe80a98f0 legacy=0x7f9fe80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.940 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.937+0000 7f9ff00f6640 1 -- 172.21.15.122:0/2983896027 shutdown_connections 2024-03-25T18:18:22.940 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.937+0000 7f9ff00f6640 1 -- 172.21.15.122:0/2983896027 >> 172.21.15.122:0/2983896027 conn(0x7f9fe80b7ea0 msgr2=0x7f9fe80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:22.941 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.937+0000 7f9ff00f6640 1 -- 172.21.15.122:0/2983896027 shutdown_connections 2024-03-25T18:18:22.941 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.937+0000 7f9ff00f6640 1 -- 172.21.15.122:0/2983896027 wait complete. 2024-03-25T18:18:22.941 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.938+0000 7f9ff00f6640 10 monclient: build_initial_monmap 2024-03-25T18:18:22.941 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.938+0000 7f9ff00f6640 10 monclient: monmap: 2024-03-25T18:18:22.941 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:18:22.941 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:22.941 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:18:22.941 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:18:22.941 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:22.941 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:18:22.941 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:18:22.941 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:18:22.941 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:18:22.941 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:22.941 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.938+0000 7f9ff00f6640 1 librados: starting msgr at 2024-03-25T18:18:22.941 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.938+0000 7f9ff00f6640 1 librados: starting objecter 2024-03-25T18:18:22.941 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.938+0000 7f9ff00f6640 1 Processor -- start 2024-03-25T18:18:22.942 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.938+0000 7f9ff00f6640 1 -- start start 2024-03-25T18:18:22.942 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.938+0000 7f9ff00f6640 1 librados: setting wanted keys 2024-03-25T18:18:22.942 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.938+0000 7f9ff00f6640 1 librados: calling monclient init 2024-03-25T18:18:22.942 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.938+0000 7f9ff00f6640 10 monclient: init 2024-03-25T18:18:22.942 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.939+0000 7f9ff00f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:22.942 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.939+0000 7f9ff00f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:22.942 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.939+0000 7f9ff00f6640 10 monclient(hunting): picked mon.noname-b con 0x7f9fe80a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:18:22.942 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.939+0000 7f9ff00f6640 10 monclient(hunting): picked mon.noname-c con 0x7f9fe813f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:18:22.942 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.939+0000 7f9ff00f6640 10 monclient(hunting): picked mon.noname-a con 0x7f9fe81430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:18:22.942 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.939+0000 7f9ff00f6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe8137ab0 con 0x7f9fe81430c0 2024-03-25T18:18:22.942 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.939+0000 7f9ff00f6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe8138bd0 con 0x7f9fe80a98f0 2024-03-25T18:18:22.942 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.939+0000 7f9ff00f6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fe8138e40 con 0x7f9fe813f710 2024-03-25T18:18:22.942 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.939+0000 7f9ff00f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:22.942 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.939+0000 7f9ff00f6640 10 monclient(hunting): authenticate will time out at 7426.205260s 2024-03-25T18:18:22.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.947+0000 7f9fee693640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f9fe81430c0 0x7f9fe81e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:48408/0 (socket says 172.21.15.122:48408) 2024-03-25T18:18:22.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.947+0000 7f9fee693640 1 -- 172.21.15.122:0/2934759588 learned_addr learned my addr 172.21.15.122:0/2934759588 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:18:22.950 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.948+0000 7f9fde7fc640 10 client.?.objecter ms_handle_connect 0x7f9fe81430c0 2024-03-25T18:18:22.950 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.949+0000 7f9fde7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:22.957 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.957+0000 7f9fde7fc640 10 client.?.objecter ms_handle_connect 0x7f9fe813f710 2024-03-25T18:18:22.958 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.957+0000 7f9fde7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:22.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.958+0000 7f9fde7fc640 10 client.?.objecter ms_handle_connect 0x7f9fe80a98f0 2024-03-25T18:18:22.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.958+0000 7f9fde7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:22.963 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.962+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245851095 0 0) 0x7f9fe8138e40 con 0x7f9fe813f710 2024-03-25T18:18:22.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.962+0000 7f9fde7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.962+0000 7f9fde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.962+0000 7f9fde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.962+0000 7f9fde7fc640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:18:22.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.962+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fcc004eb0 con 0x7f9fe813f710 2024-03-25T18:18:22.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.964+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644064735 0 0) 0x7f9fe8138bd0 con 0x7f9fe80a98f0 2024-03-25T18:18:22.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.964+0000 7f9fde7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.964+0000 7f9fde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.964+0000 7f9fde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.964+0000 7f9fde7fc640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:18:22.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.964+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe8138e40 con 0x7f9fe80a98f0 2024-03-25T18:18:22.966 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.965+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493432669 0 0) 0x7f9fe8137ab0 con 0x7f9fe81430c0 2024-03-25T18:18:22.967 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.965+0000 7f9fde7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:22.967 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.965+0000 7f9fde7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:22.967 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.965+0000 7f9fde7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:22.967 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.965+0000 7f9fde7fc640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:18:22.967 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.965+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fe8138bd0 con 0x7f9fe81430c0 2024-03-25T18:18:22.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.972+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1950029959 0 0) 0x7f9fcc004eb0 con 0x7f9fe813f710 2024-03-25T18:18:22.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.972+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fe8137ab0 con 0x7f9fe813f710 2024-03-25T18:18:22.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.972+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9fd80018f0 con 0x7f9fe813f710 2024-03-25T18:18:22.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.972+0000 7f9fde7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:22.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.972+0000 7f9fde7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:22.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.972+0000 7f9fde7fc640 10 monclient(hunting): dump: 2024-03-25T18:18:22.973 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:22.973 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:22.973 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:22.974 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:22.974 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:22.974 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:22.974 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:22.974 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:22.974 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:22.974 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:22.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.972+0000 7f9fde7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:22.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.973+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3327679653 0 0) 0x7f9fe8138e40 con 0x7f9fe80a98f0 2024-03-25T18:18:22.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.973+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fcc004eb0 con 0x7f9fe80a98f0 2024-03-25T18:18:22.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.973+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9fe4001b00 con 0x7f9fe80a98f0 2024-03-25T18:18:22.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.973+0000 7f9fde7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:22.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.973+0000 7f9fde7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:22.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.973+0000 7f9fde7fc640 10 monclient(hunting): dump: 2024-03-25T18:18:22.975 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:22.975 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:22.975 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:22.975 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:22.975 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:22.975 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:22.975 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:22.976 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:22.976 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:22.976 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:22.977 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.976+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2587662943 0 0) 0x7f9fe8138bd0 con 0x7f9fe81430c0 2024-03-25T18:18:22.977 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.976+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fe8138e40 con 0x7f9fe81430c0 2024-03-25T18:18:22.977 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.976+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9fe0001bd0 con 0x7f9fe81430c0 2024-03-25T18:18:22.977 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.976+0000 7f9fde7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:22.977 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.976+0000 7f9fde7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:22.977 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.976+0000 7f9fde7fc640 10 monclient(hunting): dump: 2024-03-25T18:18:22.977 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:22.977 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:22.978 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:22.978 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:22.978 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:22.978 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:22.978 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:22.978 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:22.978 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:22.978 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:22.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 29201532 0 0) 0x7f9fcc004eb0 con 0x7f9fe80a98f0 2024-03-25T18:18:22.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 >> v1:172.21.15.122:6790/0 conn(0x7f9fe813f710 legacy=0x7f9fe81df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 >> v1:172.21.15.122:6789/0 conn(0x7f9fe81430c0 legacy=0x7f9fe81e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:22.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9fde7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:22.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9fde7fc640 1 monclient: found mon.b 2024-03-25T18:18:22.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9fde7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:22.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9fde7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:22.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fe8139120 con 0x7f9fe80a98f0 2024-03-25T18:18:22.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9fde7fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:22.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9fde7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:22.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9fde7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:22.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9ff00f6640 5 monclient: authenticate success, global_id 4312 2024-03-25T18:18:22.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9ff00f6640 10 monclient: _renew_subs 2024-03-25T18:18:22.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9ff00f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:22.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9ff00f6640 1 -- 172.21.15.122:0/2934759588 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fe8139400 con 0x7f9fe80a98f0 2024-03-25T18:18:22.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9ff00f6640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:22.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9ff00f6640 10 monclient: _renew_subs 2024-03-25T18:18:22.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9ff00f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:22.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9ff00f6640 1 -- 172.21.15.122:0/2934759588 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fe80b7880 con 0x7f9fe80a98f0 2024-03-25T18:18:22.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.978+0000 7f9ff00f6640 1 librados: init done 2024-03-25T18:18:22.982 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.980+0000 7f9ff00f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:22.982 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.980+0000 7f9ff00f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:22.982 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.980+0000 7f9ff00f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:22.982 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:22.980+0000 7f9ff00f6640 1 -- 172.21.15.122:0/2934759588 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fe8137810 con 0x7f9fe80a98f0 2024-03-25T18:18:23.255 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.254+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fe4001ca0 con 0x7f9fe80a98f0 2024-03-25T18:18:23.255 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.254+0000 7f9fde7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:23.256 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.254+0000 7f9fdffff640 4 set_mon_vals no callback set 2024-03-25T18:18:23.256 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.255+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9fe4003a40 con 0x7f9fe80a98f0 2024-03-25T18:18:23.256 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.255+0000 7f9fde7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:23.256 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.255+0000 7f9fde7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:23.256 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.255+0000 7f9fde7fc640 10 monclient: dump: 2024-03-25T18:18:23.256 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:23.256 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:23.256 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:23.256 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:23.256 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:23.257 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:23.257 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:23.257 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:23.257 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:23.257 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:23.257 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.255+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f9fe4019a10 con 0x7f9fe80a98f0 2024-03-25T18:18:23.257 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.256+0000 7f9fed691640 10 monclient: get_auth_request con 0x7f9fcc06e110 auth_method 0 2024-03-25T18:18:23.257 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.256+0000 7f9fde7fc640 10 client.4312.objecter ms_handle_connect 0x7f9fcc06e110 2024-03-25T18:18:23.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.260+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f9fe4053620 con 0x7f9fe80a98f0 2024-03-25T18:18:23.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.260+0000 7f9fde7fc640 10 client.4312.objecter ms_dispatch 0x7f9fe8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:23.262 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.260+0000 7f9fde7fc640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:23.262 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.260+0000 7f9fde7fc640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:23.262 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.260+0000 7f9fde7fc640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-25T18:18:23.262 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.261+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9fe404f020 con 0x7f9fe80a98f0 2024-03-25T18:18:23.262 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.261+0000 7f9fde7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:23.262 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.261+0000 7f9fde7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:23.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.514+0000 7f9ff00f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:23.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.514+0000 7f9ff00f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:23.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.514+0000 7f9ff00f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:23.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.514+0000 7f9ff00f6640 1 -- 172.21.15.122:0/2934759588 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9fe80627b0 con 0x7f9fe80a98f0 2024-03-25T18:18:23.537 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.536+0000 7f9fde7fc640 1 -- 172.21.15.122:0/2934759588 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f9fe4051060 con 0x7f9fe80a98f0 2024-03-25T18:18:23.537 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.536+0000 7f9fde7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:23.537 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.536+0000 7f9fde7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:23.537 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:18:23.537 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:23.540 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.538+0000 7f9ff00f6640 10 librados: watch_flush enter 2024-03-25T18:18:23.540 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.538+0000 7f9ff00f6640 10 librados: watch_flush exit 2024-03-25T18:18:23.540 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.538+0000 7f9ff00f6640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-25T18:18:23.540 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.538+0000 7f9ff00f6640 10 client.4312.objecter successfully canceled tick 2024-03-25T18:18:23.540 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.538+0000 7f9ff00f6640 1 -- 172.21.15.122:0/2934759588 >> v1:172.21.15.204:6800/676228005 conn(0x7f9fcc06e110 legacy=0x7f9fcc0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:23.540 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.538+0000 7f9ff00f6640 10 monclient: shutdown 2024-03-25T18:18:23.540 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.538+0000 7f9ff00f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:23.540 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.538+0000 7f9ff00f6640 1 -- 172.21.15.122:0/2934759588 >> v1:172.21.15.204:6789/0 conn(0x7f9fe80a98f0 legacy=0x7f9fe8137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:23.540 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.538+0000 7f9ff00f6640 1 -- 172.21.15.122:0/2934759588 shutdown_connections 2024-03-25T18:18:23.540 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.538+0000 7f9ff00f6640 1 -- 172.21.15.122:0/2934759588 >> 172.21.15.122:0/2934759588 conn(0x7f9fe80b7ea0 msgr2=0x7f9fe8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:23.540 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.538+0000 7f9ff00f6640 1 -- 172.21.15.122:0/2934759588 shutdown_connections 2024-03-25T18:18:23.540 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.538+0000 7f9ff00f6640 1 -- 172.21.15.122:0/2934759588 wait complete. 2024-03-25T18:18:23.540 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:23.538+0000 7f9ff00f6640 1 librados: shutdown 2024-03-25T18:18:23.550 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:25.640 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:02:19.383 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:25.646 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:02:19.389 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:25.679 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:02:19.421 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:25.701 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:02:19.447 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:25.708 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:02:19.454 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:25.709 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:02:19.452 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:25.722 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:02:19.464 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:25.746 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:02:19.492 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:25.756 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:02:19.498 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:30.356 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:30.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.469+0000 7f4c2a686640 10 monclient: get_monmap_and_config 2024-03-25T18:18:30.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.469+0000 7f4c2a686640 10 monclient: build_initial_monmap 2024-03-25T18:18:30.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.469+0000 7f4c2a686640 10 monclient: monmap: 2024-03-25T18:18:30.470 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:18:30.470 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:30.470 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:18:30.470504+0000 2024-03-25T18:18:30.470 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:18:30.470504+0000 2024-03-25T18:18:30.470 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:30.470 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:30.470 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:18:30.471 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:18:30.471 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:18:30.471 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:30.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.470+0000 7f4c2a686640 1 Processor -- start 2024-03-25T18:18:30.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.470+0000 7f4c2a686640 1 -- start start 2024-03-25T18:18:30.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.470+0000 7f4c2a686640 10 monclient: init 2024-03-25T18:18:30.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.470+0000 7f4c2a686640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:30.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.470+0000 7f4c2a686640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:30.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.470+0000 7f4c2a686640 10 monclient(hunting): picked mon.noname-b con 0x7f4c240a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:18:30.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.470+0000 7f4c2a686640 10 monclient(hunting): picked mon.noname-a con 0x7f4c2413f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:18:30.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.470+0000 7f4c2a686640 10 monclient(hunting): picked mon.noname-c con 0x7f4c241430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:18:30.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.470+0000 7f4c2a686640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c24142520 con 0x7f4c2413f710 2024-03-25T18:18:30.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.470+0000 7f4c2a686640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c240b28a0 con 0x7f4c241430c0 2024-03-25T18:18:30.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.470+0000 7f4c2a686640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c240b2a30 con 0x7f4c240a98f0 2024-03-25T18:18:30.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.470+0000 7f4c2a686640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:30.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.470+0000 7f4c2a686640 10 monclient(hunting): authenticate will time out at 7433.736910s 2024-03-25T18:18:30.489 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.489+0000 7f4c28c23640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f4c241430c0 0x7f4c2414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:56420/0 (socket says 172.21.15.122:56420) 2024-03-25T18:18:30.490 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.489+0000 7f4c28c23640 1 -- 172.21.15.122:0/3339825599 learned_addr learned my addr 172.21.15.122:0/3339825599 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:18:30.500 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.499+0000 7f4c22ffd640 1 -- 172.21.15.122:0/3339825599 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581471887 0 0) 0x7f4c24142520 con 0x7f4c2413f710 2024-03-25T18:18:30.501 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.499+0000 7f4c22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:30.501 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.499+0000 7f4c22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:30.501 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.499+0000 7f4c22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:30.501 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.499+0000 7f4c22ffd640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:18:30.501 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.499+0000 7f4c22ffd640 1 -- 172.21.15.122:0/3339825599 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c0c003520 con 0x7f4c2413f710 2024-03-25T18:18:30.512 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.511+0000 7f4c22ffd640 1 -- 172.21.15.122:0/3339825599 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261870109 0 0) 0x7f4c240b28a0 con 0x7f4c241430c0 2024-03-25T18:18:30.512 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.511+0000 7f4c22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:30.512 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.511+0000 7f4c22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:30.512 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.511+0000 7f4c22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:30.512 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.511+0000 7f4c22ffd640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:18:30.512 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.511+0000 7f4c22ffd640 1 -- 172.21.15.122:0/3339825599 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c24142520 con 0x7f4c241430c0 2024-03-25T18:18:30.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.511+0000 7f4c22ffd640 1 -- 172.21.15.122:0/3339825599 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538169656 0 0) 0x7f4c240b2a30 con 0x7f4c240a98f0 2024-03-25T18:18:30.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.511+0000 7f4c22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:30.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.511+0000 7f4c22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:30.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.511+0000 7f4c22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:30.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.511+0000 7f4c22ffd640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:18:30.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.511+0000 7f4c22ffd640 1 -- 172.21.15.122:0/3339825599 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c240b28a0 con 0x7f4c240a98f0 2024-03-25T18:18:30.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c22ffd640 1 -- 172.21.15.122:0/3339825599 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3212887082 0 0) 0x7f4c0c003520 con 0x7f4c2413f710 2024-03-25T18:18:30.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c22ffd640 1 -- 172.21.15.122:0/3339825599 >> v1:172.21.15.122:6790/0 conn(0x7f4c240a98f0 legacy=0x7f4c240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c22ffd640 1 -- 172.21.15.122:0/3339825599 >> v1:172.21.15.204:6789/0 conn(0x7f4c241430c0 legacy=0x7f4c2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c22ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:30.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c22ffd640 1 monclient: found mon.noname-a 2024-03-25T18:18:30.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:30.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c22ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:18:30.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c22ffd640 1 -- 172.21.15.122:0/3339825599 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c240b44c0 con 0x7f4c2413f710 2024-03-25T18:18:30.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c22ffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:30.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c22ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:30.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:30.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c22ffd640 1 -- 172.21.15.122:0/3339825599 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4c18001690 con 0x7f4c2413f710 2024-03-25T18:18:30.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c2a686640 5 monclient: authenticate success, global_id 4317 2024-03-25T18:18:30.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c2a686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:30.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c22ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:30.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c22ffd640 10 monclient: dump: 2024-03-25T18:18:30.515 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:30.515 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:30.515 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:30.515 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:30.515 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.515 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:30.515 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:30.515 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:30.515 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:30.515 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:30.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.512+0000 7f4c2a686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:30.518 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.517+0000 7f4c22ffd640 1 -- 172.21.15.122:0/3339825599 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c180017f0 con 0x7f4c2413f710 2024-03-25T18:18:30.518 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.517+0000 7f4c22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:30.518 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.517+0000 7f4c22ffd640 1 -- 172.21.15.122:0/3339825599 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4c18001c00 con 0x7f4c2413f710 2024-03-25T18:18:30.518 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.517+0000 7f4c2a686640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:30.518 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.517+0000 7f4c2a686640 4 set_mon_vals no callback set 2024-03-25T18:18:30.518 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.517+0000 7f4c22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.518 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.517+0000 7f4c22ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:30.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.517+0000 7f4c22ffd640 10 monclient: dump: 2024-03-25T18:18:30.519 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:30.519 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:30.519 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:30.519 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:30.519 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.519 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:30.519 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:30.519 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:30.519 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:30.519 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:30.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.518+0000 7f4c2a686640 10 monclient: shutdown 2024-03-25T18:18:30.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.518+0000 7f4c2a686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:30.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.518+0000 7f4c2a686640 1 -- 172.21.15.122:0/3339825599 >> v1:172.21.15.122:6789/0 conn(0x7f4c2413f710 legacy=0x7f4c241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.518+0000 7f4c2a686640 1 -- 172.21.15.122:0/3339825599 shutdown_connections 2024-03-25T18:18:30.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.518+0000 7f4c2a686640 1 -- 172.21.15.122:0/3339825599 >> 172.21.15.122:0/3339825599 conn(0x7f4c240b7ea0 msgr2=0x7f4c240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:30.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.518+0000 7f4c2a686640 1 -- 172.21.15.122:0/3339825599 shutdown_connections 2024-03-25T18:18:30.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.518+0000 7f4c2a686640 1 -- 172.21.15.122:0/3339825599 wait complete. 2024-03-25T18:18:30.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.518+0000 7f4c2a686640 10 monclient: build_initial_monmap 2024-03-25T18:18:30.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.518+0000 7f4c2a686640 10 monclient: monmap: 2024-03-25T18:18:30.522 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:18:30.522 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:30.522 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:18:30.522 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:18:30.522 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:30.523 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:18:30.523 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:18:30.523 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:18:30.523 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:18:30.523 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:30.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.519+0000 7f4c2a686640 1 librados: starting msgr at 2024-03-25T18:18:30.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.519+0000 7f4c2a686640 1 librados: starting objecter 2024-03-25T18:18:30.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.519+0000 7f4c2a686640 1 Processor -- start 2024-03-25T18:18:30.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.519+0000 7f4c2a686640 1 -- start start 2024-03-25T18:18:30.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.519+0000 7f4c2a686640 1 librados: setting wanted keys 2024-03-25T18:18:30.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.519+0000 7f4c2a686640 1 librados: calling monclient init 2024-03-25T18:18:30.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.519+0000 7f4c2a686640 10 monclient: init 2024-03-25T18:18:30.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.520+0000 7f4c2a686640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:30.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.520+0000 7f4c2a686640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:30.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.520+0000 7f4c2a686640 10 monclient(hunting): picked mon.noname-c con 0x7f4c240a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:18:30.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.520+0000 7f4c2a686640 10 monclient(hunting): picked mon.noname-b con 0x7f4c2413f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:18:30.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.520+0000 7f4c2a686640 10 monclient(hunting): picked mon.noname-a con 0x7f4c241430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:18:30.524 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.520+0000 7f4c2a686640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c241d82d0 con 0x7f4c241430c0 2024-03-25T18:18:30.524 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.520+0000 7f4c2a686640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c240abb90 con 0x7f4c2413f710 2024-03-25T18:18:30.524 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.520+0000 7f4c2a686640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c241d76a0 con 0x7f4c240a98f0 2024-03-25T18:18:30.524 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.520+0000 7f4c2a686640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:30.524 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.520+0000 7f4c2a686640 10 monclient(hunting): authenticate will time out at 7433.786930s 2024-03-25T18:18:30.537 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.536+0000 7f4c28c23640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4c241430c0 0x7f4c24134010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:48954/0 (socket says 172.21.15.122:48954) 2024-03-25T18:18:30.537 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.536+0000 7f4c28c23640 1 -- 172.21.15.122:0/2784006830 learned_addr learned my addr 172.21.15.122:0/2784006830 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:18:30.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.538+0000 7f4c20ff9640 10 client.?.objecter ms_handle_connect 0x7f4c241430c0 2024-03-25T18:18:30.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.538+0000 7f4c20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:30.542 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.541+0000 7f4c20ff9640 10 client.?.objecter ms_handle_connect 0x7f4c240a98f0 2024-03-25T18:18:30.542 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.541+0000 7f4c20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:30.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.545+0000 7f4c20ff9640 10 client.?.objecter ms_handle_connect 0x7f4c2413f710 2024-03-25T18:18:30.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.545+0000 7f4c20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:30.550 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.549+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3953845306 0 0) 0x7f4c241d82d0 con 0x7f4c241430c0 2024-03-25T18:18:30.550 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.549+0000 7f4c20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:30.551 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.549+0000 7f4c20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:30.551 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.549+0000 7f4c20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:30.551 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.549+0000 7f4c20ff9640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:18:30.551 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.550+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c10004eb0 con 0x7f4c241430c0 2024-03-25T18:18:30.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.552+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2456835936 0 0) 0x7f4c240abb90 con 0x7f4c2413f710 2024-03-25T18:18:30.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.552+0000 7f4c20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:30.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.552+0000 7f4c20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:30.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.552+0000 7f4c20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:30.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.552+0000 7f4c20ff9640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:18:30.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.552+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c241d82d0 con 0x7f4c2413f710 2024-03-25T18:18:30.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.556+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 530773419 0 0) 0x7f4c241d76a0 con 0x7f4c240a98f0 2024-03-25T18:18:30.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.556+0000 7f4c20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:30.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.556+0000 7f4c20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:30.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.556+0000 7f4c20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:30.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.556+0000 7f4c20ff9640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:18:30.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.556+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c240abb90 con 0x7f4c240a98f0 2024-03-25T18:18:30.560 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.559+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 847165636 0 0) 0x7f4c241d82d0 con 0x7f4c2413f710 2024-03-25T18:18:30.560 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.559+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c241d76a0 con 0x7f4c2413f710 2024-03-25T18:18:30.560 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.559+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4c18001b00 con 0x7f4c2413f710 2024-03-25T18:18:30.560 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.559+0000 7f4c20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.560 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.559+0000 7f4c20ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:30.560 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.559+0000 7f4c20ff9640 10 monclient(hunting): dump: 2024-03-25T18:18:30.560 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:30.561 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:30.561 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:30.561 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:30.561 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.561 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:30.561 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:30.561 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:30.561 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:30.561 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:30.561 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.559+0000 7f4c20ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:30.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.567+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 389843534 0 0) 0x7f4c240abb90 con 0x7f4c240a98f0 2024-03-25T18:18:30.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.567+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c241d82d0 con 0x7f4c240a98f0 2024-03-25T18:18:30.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.567+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4c14001830 con 0x7f4c240a98f0 2024-03-25T18:18:30.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.567+0000 7f4c20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.567+0000 7f4c20ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:30.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.567+0000 7f4c20ff9640 10 monclient(hunting): dump: 2024-03-25T18:18:30.568 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:30.568 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:30.568 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:30.568 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:30.568 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.568 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:30.568 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:30.568 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:30.569 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:30.569 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:30.570 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.567+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1697317741 0 0) 0x7f4c10004eb0 con 0x7f4c241430c0 2024-03-25T18:18:30.570 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.567+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c240abb90 con 0x7f4c241430c0 2024-03-25T18:18:30.570 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.567+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4c1c001c70 con 0x7f4c241430c0 2024-03-25T18:18:30.570 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.567+0000 7f4c20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.570 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.567+0000 7f4c20ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:30.570 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.567+0000 7f4c20ff9640 10 monclient(hunting): dump: 2024-03-25T18:18:30.570 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:30.571 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:30.571 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:30.571 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:30.571 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.571 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:30.571 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:30.571 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:30.571 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:30.571 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:30.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1995428501 0 0) 0x7f4c241d76a0 con 0x7f4c2413f710 2024-03-25T18:18:30.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 >> v1:172.21.15.122:6790/0 conn(0x7f4c240a98f0 legacy=0x7f4c241385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 >> v1:172.21.15.122:6789/0 conn(0x7f4c241430c0 legacy=0x7f4c24134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c20ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:18:30.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c20ff9640 1 monclient: found mon.b 2024-03-25T18:18:30.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c20ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:30.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c20ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:30.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c24134720 con 0x7f4c2413f710 2024-03-25T18:18:30.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c20ff9640 10 monclient: _finish_auth 0 2024-03-25T18:18:30.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c20ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:18:30.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:30.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c2a686640 5 monclient: authenticate success, global_id 4318 2024-03-25T18:18:30.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c2a686640 10 monclient: _renew_subs 2024-03-25T18:18:30.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c2a686640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:30.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c2a686640 1 -- 172.21.15.122:0/2784006830 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c24134970 con 0x7f4c2413f710 2024-03-25T18:18:30.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c2a686640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:30.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c2a686640 10 monclient: _renew_subs 2024-03-25T18:18:30.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c2a686640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:30.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c2a686640 1 -- 172.21.15.122:0/2784006830 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c24137a10 con 0x7f4c2413f710 2024-03-25T18:18:30.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.568+0000 7f4c2a686640 1 librados: init done 2024-03-25T18:18:30.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.569+0000 7f4c2a686640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:30.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.569+0000 7f4c2a686640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:30.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.569+0000 7f4c2a686640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:30.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.569+0000 7f4c2a686640 1 -- 172.21.15.122:0/2784006830 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c24074cc0 con 0x7f4c2413f710 2024-03-25T18:18:30.845 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.844+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c18001ca0 con 0x7f4c2413f710 2024-03-25T18:18:30.845 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.844+0000 7f4c20ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:30.845 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.844+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4c18003a40 con 0x7f4c2413f710 2024-03-25T18:18:30.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.844+0000 7f4c20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.844+0000 7f4c20ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:30.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.844+0000 7f4c20ff9640 10 monclient: dump: 2024-03-25T18:18:30.846 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:30.846 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:30.846 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:30.846 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:30.846 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.846 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:30.846 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:30.846 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:30.846 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:30.846 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:30.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.844+0000 7f4c227fc640 4 set_mon_vals no callback set 2024-03-25T18:18:30.847 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.845+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f4c18019a10 con 0x7f4c2413f710 2024-03-25T18:18:30.847 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.846+0000 7f4c23fff640 10 monclient: get_auth_request con 0x7f4c1006e0f0 auth_method 0 2024-03-25T18:18:30.847 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.846+0000 7f4c20ff9640 10 client.4318.objecter ms_handle_connect 0x7f4c1006e0f0 2024-03-25T18:18:30.851 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.850+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f4c180535a0 con 0x7f4c2413f710 2024-03-25T18:18:30.851 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.850+0000 7f4c20ff9640 10 client.4318.objecter ms_dispatch 0x7f4c24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:30.851 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.850+0000 7f4c20ff9640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:30.851 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.850+0000 7f4c20ff9640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:30.851 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.850+0000 7f4c20ff9640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-25T18:18:30.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.850+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4c1804f0b0 con 0x7f4c2413f710 2024-03-25T18:18:30.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.850+0000 7f4c20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:30.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:30.850+0000 7f4c20ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:31.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.073+0000 7f4c2a686640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:31.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.073+0000 7f4c2a686640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:31.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.073+0000 7f4c2a686640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:31.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.073+0000 7f4c2a686640 1 -- 172.21.15.122:0/2784006830 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4c241d7830 con 0x7f4c2413f710 2024-03-25T18:18:31.146 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.145+0000 7f4c20ff9640 1 -- 172.21.15.122:0/2784006830 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f4c18050fe0 con 0x7f4c2413f710 2024-03-25T18:18:31.146 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.145+0000 7f4c20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:31.147 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.145+0000 7f4c20ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:31.147 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:18:31.147 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:31.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.147+0000 7f4c2a686640 10 librados: watch_flush enter 2024-03-25T18:18:31.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.147+0000 7f4c2a686640 10 librados: watch_flush exit 2024-03-25T18:18:31.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.147+0000 7f4c2a686640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-25T18:18:31.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.148+0000 7f4c2a686640 10 client.4318.objecter successfully canceled tick 2024-03-25T18:18:31.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.148+0000 7f4c2a686640 1 -- 172.21.15.122:0/2784006830 >> v1:172.21.15.204:6800/676228005 conn(0x7f4c1006e0f0 legacy=0x7f4c100705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:31.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.148+0000 7f4c2a686640 10 monclient: shutdown 2024-03-25T18:18:31.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.148+0000 7f4c2a686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:31.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.148+0000 7f4c2a686640 1 -- 172.21.15.122:0/2784006830 >> v1:172.21.15.204:6789/0 conn(0x7f4c2413f710 legacy=0x7f4c241d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:31.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.148+0000 7f4c2a686640 1 -- 172.21.15.122:0/2784006830 shutdown_connections 2024-03-25T18:18:31.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.148+0000 7f4c2a686640 1 -- 172.21.15.122:0/2784006830 >> 172.21.15.122:0/2784006830 conn(0x7f4c240b7ea0 msgr2=0x7f4c24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:31.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.148+0000 7f4c2a686640 1 -- 172.21.15.122:0/2784006830 shutdown_connections 2024-03-25T18:18:31.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.148+0000 7f4c2a686640 1 -- 172.21.15.122:0/2784006830 wait complete. 2024-03-25T18:18:31.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:31.148+0000 7f4c2a686640 1 librados: shutdown 2024-03-25T18:18:31.161 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:37.968 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:38.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.086+0000 7fd7925e6640 10 monclient: get_monmap_and_config 2024-03-25T18:18:38.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.086+0000 7fd7925e6640 10 monclient: build_initial_monmap 2024-03-25T18:18:38.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.086+0000 7fd7925e6640 10 monclient: monmap: 2024-03-25T18:18:38.087 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:18:38.087 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:38.087 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:18:38.086847+0000 2024-03-25T18:18:38.087 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:18:38.086847+0000 2024-03-25T18:18:38.087 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:38.087 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:38.087 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:18:38.087 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:18:38.087 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:18:38.087 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:38.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.087+0000 7fd7925e6640 1 Processor -- start 2024-03-25T18:18:38.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.087+0000 7fd7925e6640 1 -- start start 2024-03-25T18:18:38.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.087+0000 7fd7925e6640 10 monclient: init 2024-03-25T18:18:38.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.088+0000 7fd7925e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:38.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.088+0000 7fd7925e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:38.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.088+0000 7fd7925e6640 10 monclient(hunting): picked mon.noname-a con 0x7fd78c13fc60 addr v1:172.21.15.122:6789/0 2024-03-25T18:18:38.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.088+0000 7fd7925e6640 10 monclient(hunting): picked mon.noname-c con 0x7fd78c070570 addr v1:172.21.15.204:6789/0 2024-03-25T18:18:38.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.088+0000 7fd7925e6640 10 monclient(hunting): picked mon.noname-b con 0x7fd78c0727a0 addr v1:172.21.15.122:6790/0 2024-03-25T18:18:38.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.088+0000 7fd7925e6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd78c13c9f0 con 0x7fd78c13fc60 2024-03-25T18:18:38.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.088+0000 7fd7925e6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd78c06fac0 con 0x7fd78c070570 2024-03-25T18:18:38.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.088+0000 7fd7925e6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd78c0715d0 con 0x7fd78c0727a0 2024-03-25T18:18:38.090 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.088+0000 7fd7925e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:38.090 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.088+0000 7fd7925e6640 10 monclient(hunting): authenticate will time out at 7441.353597s 2024-03-25T18:18:38.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.093+0000 7fd790b83640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7fd78c0727a0 0x7fd78c14a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:54084/0 (socket says 172.21.15.122:54084) 2024-03-25T18:18:38.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.093+0000 7fd790b83640 1 -- 172.21.15.122:0/3814354019 learned_addr learned my addr 172.21.15.122:0/3814354019 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:18:38.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.099+0000 7fd78affd640 1 -- 172.21.15.122:0/3814354019 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2894017448 0 0) 0x7fd78c0715d0 con 0x7fd78c0727a0 2024-03-25T18:18:38.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.099+0000 7fd78affd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:38.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.099+0000 7fd78affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:38.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.099+0000 7fd78affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:38.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.099+0000 7fd78affd640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:18:38.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.099+0000 7fd78affd640 1 -- 172.21.15.122:0/3814354019 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd774003520 con 0x7fd78c0727a0 2024-03-25T18:18:38.115 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.115+0000 7fd78affd640 1 -- 172.21.15.122:0/3814354019 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743744861 0 0) 0x7fd78c06fac0 con 0x7fd78c070570 2024-03-25T18:18:38.116 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.115+0000 7fd78affd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:38.116 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.115+0000 7fd78affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:38.116 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.115+0000 7fd78affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:38.116 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.115+0000 7fd78affd640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:18:38.116 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.115+0000 7fd78affd640 1 -- 172.21.15.122:0/3814354019 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd78c0715d0 con 0x7fd78c070570 2024-03-25T18:18:38.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd78affd640 1 -- 172.21.15.122:0/3814354019 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1088333188 0 0) 0x7fd774003520 con 0x7fd78c0727a0 2024-03-25T18:18:38.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd78affd640 1 -- 172.21.15.122:0/3814354019 >> v1:172.21.15.204:6789/0 conn(0x7fd78c070570 legacy=0x7fd78c072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:38.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd78affd640 1 -- 172.21.15.122:0/3814354019 >> v1:172.21.15.122:6789/0 conn(0x7fd78c13fc60 legacy=0x7fd78c06f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:38.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd78affd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:38.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd78affd640 1 monclient: found mon.noname-b 2024-03-25T18:18:38.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd78affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:38.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd78affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:18:38.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd78affd640 1 -- 172.21.15.122:0/3814354019 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78c0718b0 con 0x7fd78c0727a0 2024-03-25T18:18:38.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd78affd640 10 monclient: _finish_auth 0 2024-03-25T18:18:38.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd78affd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:38.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd78affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:38.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd78affd640 1 -- 172.21.15.122:0/3814354019 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd784001a10 con 0x7fd78c0727a0 2024-03-25T18:18:38.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd7925e6640 5 monclient: authenticate success, global_id 4304 2024-03-25T18:18:38.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd7925e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:38.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd78affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:38.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd78affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:38.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd78affd640 10 monclient: dump: 2024-03-25T18:18:38.118 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:38.118 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:38.119 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:38.119 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:38.119 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:38.119 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:38.119 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:38.119 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:38.119 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:38.119 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:38.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.116+0000 7fd7925e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:38.135 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.135+0000 7fd78affd640 1 -- 172.21.15.122:0/3814354019 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd784001b70 con 0x7fd78c0727a0 2024-03-25T18:18:38.135 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.135+0000 7fd78affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:38.135 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.135+0000 7fd78affd640 1 -- 172.21.15.122:0/3814354019 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd784001f80 con 0x7fd78c0727a0 2024-03-25T18:18:38.135 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.135+0000 7fd78affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:38.135 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.135+0000 7fd78affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:38.135 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.135+0000 7fd78affd640 10 monclient: dump: 2024-03-25T18:18:38.135 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:38.135 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:38.136 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:38.136 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:38.136 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:38.136 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:38.136 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:38.136 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:38.136 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:38.136 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:38.136 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.135+0000 7fd7925e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:38.136 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.135+0000 7fd7925e6640 4 set_mon_vals no callback set 2024-03-25T18:18:38.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.136+0000 7fd7925e6640 10 monclient: shutdown 2024-03-25T18:18:38.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.136+0000 7fd7925e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:38.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.136+0000 7fd7925e6640 1 -- 172.21.15.122:0/3814354019 >> v1:172.21.15.122:6790/0 conn(0x7fd78c0727a0 legacy=0x7fd78c14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:38.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.136+0000 7fd7925e6640 1 -- 172.21.15.122:0/3814354019 shutdown_connections 2024-03-25T18:18:38.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.136+0000 7fd7925e6640 1 -- 172.21.15.122:0/3814354019 >> 172.21.15.122:0/3814354019 conn(0x7fd78c0b4a80 msgr2=0x7fd78c0b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:38.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.136+0000 7fd7925e6640 1 -- 172.21.15.122:0/3814354019 shutdown_connections 2024-03-25T18:18:38.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.136+0000 7fd7925e6640 1 -- 172.21.15.122:0/3814354019 wait complete. 2024-03-25T18:18:38.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.137+0000 7fd7925e6640 10 monclient: build_initial_monmap 2024-03-25T18:18:38.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.137+0000 7fd7925e6640 10 monclient: monmap: 2024-03-25T18:18:38.139 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:18:38.139 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:38.139 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:18:38.139 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:18:38.139 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:38.139 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:18:38.139 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:18:38.139 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:18:38.140 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:18:38.140 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:38.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.137+0000 7fd7925e6640 1 librados: starting msgr at 2024-03-25T18:18:38.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.137+0000 7fd7925e6640 1 librados: starting objecter 2024-03-25T18:18:38.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.137+0000 7fd7925e6640 1 Processor -- start 2024-03-25T18:18:38.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.137+0000 7fd7925e6640 1 -- start start 2024-03-25T18:18:38.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.137+0000 7fd7925e6640 1 librados: setting wanted keys 2024-03-25T18:18:38.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.137+0000 7fd7925e6640 1 librados: calling monclient init 2024-03-25T18:18:38.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.137+0000 7fd7925e6640 10 monclient: init 2024-03-25T18:18:38.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.138+0000 7fd7925e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:38.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.138+0000 7fd7925e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:38.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.138+0000 7fd7925e6640 10 monclient(hunting): picked mon.noname-a con 0x7fd78c070570 addr v1:172.21.15.122:6789/0 2024-03-25T18:18:38.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.138+0000 7fd7925e6640 10 monclient(hunting): picked mon.noname-c con 0x7fd78c0727a0 addr v1:172.21.15.122:6790/0 2024-03-25T18:18:38.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.138+0000 7fd7925e6640 10 monclient(hunting): picked mon.noname-b con 0x7fd78c13fc60 addr v1:172.21.15.204:6789/0 2024-03-25T18:18:38.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.138+0000 7fd7925e6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd78c134f60 con 0x7fd78c070570 2024-03-25T18:18:38.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.138+0000 7fd7925e6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd78c1351d0 con 0x7fd78c13fc60 2024-03-25T18:18:38.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.138+0000 7fd7925e6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd78c1daec0 con 0x7fd78c0727a0 2024-03-25T18:18:38.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.138+0000 7fd7925e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:38.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.138+0000 7fd7925e6640 10 monclient(hunting): authenticate will time out at 7441.403499s 2024-03-25T18:18:38.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.150+0000 7fd78bfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd78c070570 0x7fd78c1db3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:53824/0 (socket says 172.21.15.122:53824) 2024-03-25T18:18:38.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.151+0000 7fd78bfff640 1 -- 172.21.15.122:0/4206454375 learned_addr learned my addr 172.21.15.122:0/4206454375 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:18:38.155 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.155+0000 7fd788ff9640 10 client.?.objecter ms_handle_connect 0x7fd78c070570 2024-03-25T18:18:38.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.155+0000 7fd788ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:38.158 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.158+0000 7fd788ff9640 10 client.?.objecter ms_handle_connect 0x7fd78c0727a0 2024-03-25T18:18:38.158 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.158+0000 7fd788ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:38.159 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.159+0000 7fd788ff9640 10 client.?.objecter ms_handle_connect 0x7fd78c13fc60 2024-03-25T18:18:38.159 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.159+0000 7fd788ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:38.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.166+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208088755 0 0) 0x7fd78c134f60 con 0x7fd78c070570 2024-03-25T18:18:38.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.166+0000 7fd788ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:38.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.166+0000 7fd788ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:38.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.166+0000 7fd788ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:38.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.166+0000 7fd788ff9640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:18:38.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.166+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd778004eb0 con 0x7fd78c070570 2024-03-25T18:18:38.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.168+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764219634 0 0) 0x7fd78c1daec0 con 0x7fd78c0727a0 2024-03-25T18:18:38.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.168+0000 7fd788ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:38.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.168+0000 7fd788ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:38.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.168+0000 7fd788ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:38.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.168+0000 7fd788ff9640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:18:38.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.168+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd78c134f60 con 0x7fd78c0727a0 2024-03-25T18:18:38.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.176+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429094661 0 0) 0x7fd78c1351d0 con 0x7fd78c13fc60 2024-03-25T18:18:38.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.176+0000 7fd788ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:38.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.176+0000 7fd788ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:38.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.176+0000 7fd788ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:38.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.176+0000 7fd788ff9640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:18:38.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.176+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd78c1daec0 con 0x7fd78c13fc60 2024-03-25T18:18:38.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.180+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2957050675 0 0) 0x7fd778004eb0 con 0x7fd78c070570 2024-03-25T18:18:38.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.180+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd78c1351d0 con 0x7fd78c070570 2024-03-25T18:18:38.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.180+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd77c0019d0 con 0x7fd78c070570 2024-03-25T18:18:38.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.180+0000 7fd788ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:38.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.180+0000 7fd788ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:38.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.180+0000 7fd788ff9640 10 monclient(hunting): dump: 2024-03-25T18:18:38.181 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:38.181 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:38.181 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:38.181 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:38.181 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:38.181 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:38.182 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:38.182 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:38.182 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:38.182 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:38.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.181+0000 7fd788ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:38.186 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.186+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 748137172 0 0) 0x7fd78c134f60 con 0x7fd78c0727a0 2024-03-25T18:18:38.186 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.186+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd778004eb0 con 0x7fd78c0727a0 2024-03-25T18:18:38.186 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.186+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd780001910 con 0x7fd78c0727a0 2024-03-25T18:18:38.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.186+0000 7fd788ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:38.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.186+0000 7fd788ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:38.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.186+0000 7fd788ff9640 10 monclient(hunting): dump: 2024-03-25T18:18:38.187 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:38.187 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:38.187 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:38.187 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:38.187 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:38.187 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:38.187 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:38.187 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:38.187 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:38.187 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:38.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 169474848 0 0) 0x7fd78c1351d0 con 0x7fd78c070570 2024-03-25T18:18:38.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 >> v1:172.21.15.122:6790/0 conn(0x7fd78c0727a0 legacy=0x7fd78c12eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:38.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 >> v1:172.21.15.204:6789/0 conn(0x7fd78c13fc60 legacy=0x7fd78c132ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:38.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd788ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:18:38.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd788ff9640 1 monclient: found mon.a 2024-03-25T18:18:38.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd788ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:38.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd788ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:18:38.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd78c1dbaf0 con 0x7fd78c070570 2024-03-25T18:18:38.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd788ff9640 10 monclient: _finish_auth 0 2024-03-25T18:18:38.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd788ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:18:38.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd788ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:38.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd7925e6640 5 monclient: authenticate success, global_id 4326 2024-03-25T18:18:38.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd7925e6640 10 monclient: _renew_subs 2024-03-25T18:18:38.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd7925e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:18:38.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd7925e6640 1 -- 172.21.15.122:0/4206454375 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd78c1dbce0 con 0x7fd78c070570 2024-03-25T18:18:38.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd7925e6640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:38.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd7925e6640 10 monclient: _renew_subs 2024-03-25T18:18:38.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd7925e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:18:38.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd7925e6640 1 -- 172.21.15.122:0/4206454375 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd78c140690 con 0x7fd78c070570 2024-03-25T18:18:38.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.190+0000 7fd7925e6640 1 librados: init done 2024-03-25T18:18:38.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.192+0000 7fd7925e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:38.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.192+0000 7fd7925e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:38.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.192+0000 7fd7925e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:18:38.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.192+0000 7fd7925e6640 1 -- 172.21.15.122:0/4206454375 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd78c1db050 con 0x7fd78c070570 2024-03-25T18:18:38.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.491+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd77c003610 con 0x7fd78c070570 2024-03-25T18:18:38.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.491+0000 7fd788ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:38.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.491+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd77c003880 con 0x7fd78c070570 2024-03-25T18:18:38.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.491+0000 7fd788ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:38.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.491+0000 7fd788ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:38.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.491+0000 7fd788ff9640 10 monclient: dump: 2024-03-25T18:18:38.492 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:38.493 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:38.493 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:38.493 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:38.493 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:38.493 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:38.493 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:38.493 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:38.493 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:38.493 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:38.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.491+0000 7fd78a7fc640 4 set_mon_vals no callback set 2024-03-25T18:18:38.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.492+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fd77c003aa0 con 0x7fd78c070570 2024-03-25T18:18:38.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.493+0000 7fd78b7fe640 10 monclient: get_auth_request con 0x7fd77806df70 auth_method 0 2024-03-25T18:18:38.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.493+0000 7fd788ff9640 10 client.4326.objecter ms_handle_connect 0x7fd77806df70 2024-03-25T18:18:38.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.497+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fd77c054680 con 0x7fd78c070570 2024-03-25T18:18:38.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.497+0000 7fd788ff9640 10 client.4326.objecter ms_dispatch 0x7fd78c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:38.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.497+0000 7fd788ff9640 3 client.4326.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:38.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.497+0000 7fd788ff9640 3 client.4326.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:38.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.498+0000 7fd788ff9640 20 client.4326.objecter dump_active .. 0 homeless 2024-03-25T18:18:38.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.498+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd77c050190 con 0x7fd78c070570 2024-03-25T18:18:38.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.498+0000 7fd788ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:38.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.498+0000 7fd788ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:38.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.737+0000 7fd7925e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:38.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.737+0000 7fd7925e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:38.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.737+0000 7fd7925e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:18:38.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.737+0000 7fd7925e6640 1 -- 172.21.15.122:0/4206454375 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd78c0627b0 con 0x7fd78c070570 2024-03-25T18:18:38.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.803+0000 7fd788ff9640 1 -- 172.21.15.122:0/4206454375 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fd77c0520c0 con 0x7fd78c070570 2024-03-25T18:18:38.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.803+0000 7fd788ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:38.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.803+0000 7fd788ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:38.804 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:18:38.804 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:38.807 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.806+0000 7fd7925e6640 10 librados: watch_flush enter 2024-03-25T18:18:38.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.806+0000 7fd7925e6640 10 librados: watch_flush exit 2024-03-25T18:18:38.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.806+0000 7fd7925e6640 20 client.4326.objecter shutdown clearing up homeless session... 2024-03-25T18:18:38.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.806+0000 7fd7925e6640 10 client.4326.objecter successfully canceled tick 2024-03-25T18:18:38.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.806+0000 7fd7925e6640 1 -- 172.21.15.122:0/4206454375 >> v1:172.21.15.204:6800/676228005 conn(0x7fd77806df70 legacy=0x7fd778070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:38.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.806+0000 7fd7925e6640 10 monclient: shutdown 2024-03-25T18:18:38.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.806+0000 7fd7925e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:38.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.806+0000 7fd7925e6640 1 -- 172.21.15.122:0/4206454375 >> v1:172.21.15.122:6789/0 conn(0x7fd78c070570 legacy=0x7fd78c1db3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:38.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.806+0000 7fd7925e6640 1 -- 172.21.15.122:0/4206454375 shutdown_connections 2024-03-25T18:18:38.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.806+0000 7fd7925e6640 1 -- 172.21.15.122:0/4206454375 >> 172.21.15.122:0/4206454375 conn(0x7fd78c0b4a80 msgr2=0x7fd78c0b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:38.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.806+0000 7fd7925e6640 1 -- 172.21.15.122:0/4206454375 shutdown_connections 2024-03-25T18:18:38.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.806+0000 7fd7925e6640 1 -- 172.21.15.122:0/4206454375 wait complete. 2024-03-25T18:18:38.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:38.806+0000 7fd7925e6640 1 librados: shutdown 2024-03-25T18:18:38.819 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:45.623 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:45.732 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.731+0000 7f87291af640 10 monclient: get_monmap_and_config 2024-03-25T18:18:45.732 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.731+0000 7f87291af640 10 monclient: build_initial_monmap 2024-03-25T18:18:45.732 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.731+0000 7f87291af640 10 monclient: monmap: 2024-03-25T18:18:45.732 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:18:45.732 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:45.732 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:18:45.732371+0000 2024-03-25T18:18:45.732 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:18:45.732371+0000 2024-03-25T18:18:45.732 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:45.732 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:45.733 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:18:45.733 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:18:45.733 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:18:45.733 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:45.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.731+0000 7f87291af640 1 Processor -- start 2024-03-25T18:18:45.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.732+0000 7f87291af640 1 -- start start 2024-03-25T18:18:45.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.732+0000 7f87291af640 10 monclient: init 2024-03-25T18:18:45.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.732+0000 7f87291af640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:45.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.732+0000 7f87291af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:45.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.732+0000 7f87291af640 10 monclient(hunting): picked mon.noname-b con 0x7f87240b4d60 addr v1:172.21.15.122:6790/0 2024-03-25T18:18:45.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.732+0000 7f87291af640 10 monclient(hunting): picked mon.noname-a con 0x7f87240b1730 addr v1:172.21.15.122:6789/0 2024-03-25T18:18:45.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.732+0000 7f87291af640 10 monclient(hunting): picked mon.noname-c con 0x7f872414be90 addr v1:172.21.15.204:6789/0 2024-03-25T18:18:45.735 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.732+0000 7f87291af640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87240af940 con 0x7f87240b1730 2024-03-25T18:18:45.735 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.732+0000 7f87291af640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87240b0a70 con 0x7f872414be90 2024-03-25T18:18:45.735 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.732+0000 7f87291af640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87240b0d50 con 0x7f87240b4d60 2024-03-25T18:18:45.735 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.732+0000 7f87291af640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:45.735 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.732+0000 7f87291af640 10 monclient(hunting): authenticate will time out at 7448.998578s 2024-03-25T18:18:45.735 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.733+0000 7f8722d9d640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f87240b4d60 0x7f872413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:45780/0 (socket says 172.21.15.122:45780) 2024-03-25T18:18:45.735 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.733+0000 7f8722d9d640 1 -- 172.21.15.122:0/4275522081 learned_addr learned my addr 172.21.15.122:0/4275522081 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:18:45.742 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:02:39.485 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:45.752 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:02:39.494 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:45.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.765+0000 7f872259c640 1 -- 172.21.15.122:0/4275522081 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671072414 0 0) 0x7f87240af940 con 0x7f87240b1730 2024-03-25T18:18:45.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.765+0000 7f872259c640 10 monclient(hunting): _negotiate 2024-03-25T18:18:45.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.765+0000 7f872259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:45.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.765+0000 7f872259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:45.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.765+0000 7f872259c640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:18:45.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.765+0000 7f872259c640 1 -- 172.21.15.122:0/4275522081 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8710003520 con 0x7f87240b1730 2024-03-25T18:18:45.788 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:02:39.530 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:45.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f872259c640 1 -- 172.21.15.122:0/4275522081 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2013361783 0 0) 0x7f8710003520 con 0x7f87240b1730 2024-03-25T18:18:45.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f872259c640 1 -- 172.21.15.122:0/4275522081 >> v1:172.21.15.122:6790/0 conn(0x7f87240b4d60 legacy=0x7f872413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f872259c640 1 -- 172.21.15.122:0/4275522081 >> v1:172.21.15.204:6789/0 conn(0x7f872414be90 legacy=0x7f872414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f872259c640 10 monclient: _finish_hunting 0 2024-03-25T18:18:45.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f872259c640 1 monclient: found mon.noname-a 2024-03-25T18:18:45.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f872259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:45.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f872259c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:18:45.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f872259c640 1 -- 172.21.15.122:0/4275522081 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87240b1030 con 0x7f87240b1730 2024-03-25T18:18:45.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f872259c640 10 monclient: _finish_auth 0 2024-03-25T18:18:45.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f872259c640 10 monclient: _check_auth_tickets 2024-03-25T18:18:45.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f872259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:45.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f872259c640 1 -- 172.21.15.122:0/4275522081 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f870c001690 con 0x7f87240b1730 2024-03-25T18:18:45.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f87291af640 5 monclient: authenticate success, global_id 4329 2024-03-25T18:18:45.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f87291af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:45.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f872259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:45.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f872259c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:45.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f872259c640 10 monclient: dump: 2024-03-25T18:18:45.791 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:45.791 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:45.791 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:45.791 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:45.792 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:45.792 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:45.792 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:45.792 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:45.792 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:45.792 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:45.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.788+0000 7f87291af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:45.792 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:02:39.538 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:45.798 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:02:39.541 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:45.801 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:02:39.547 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:45.813 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:02:39.555 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:45.820 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:02:39.566 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:45.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.827+0000 7f872259c640 1 -- 172.21.15.122:0/4275522081 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f870c0017f0 con 0x7f87240b1730 2024-03-25T18:18:45.829 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.827+0000 7f872259c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:45.829 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.827+0000 7f872259c640 1 -- 172.21.15.122:0/4275522081 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f870c001c00 con 0x7f87240b1730 2024-03-25T18:18:45.829 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.827+0000 7f87291af640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:45.829 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.827+0000 7f87291af640 4 set_mon_vals no callback set 2024-03-25T18:18:45.829 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.827+0000 7f872259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:45.829 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.827+0000 7f872259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:45.829 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.827+0000 7f872259c640 10 monclient: dump: 2024-03-25T18:18:45.829 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:45.829 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:45.829 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:45.829 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:45.829 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:45.829 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:45.829 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:45.830 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:45.830 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:45.830 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:45.831 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.828+0000 7f87291af640 10 monclient: shutdown 2024-03-25T18:18:45.831 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.828+0000 7f87291af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:45.831 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.828+0000 7f87291af640 1 -- 172.21.15.122:0/4275522081 >> v1:172.21.15.122:6789/0 conn(0x7f87240b1730 legacy=0x7f87240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.831 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.828+0000 7f87291af640 1 -- 172.21.15.122:0/4275522081 shutdown_connections 2024-03-25T18:18:45.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.828+0000 7f87291af640 1 -- 172.21.15.122:0/4275522081 >> 172.21.15.122:0/4275522081 conn(0x7f87240617a0 msgr2=0x7f8724061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:45.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.828+0000 7f87291af640 1 -- 172.21.15.122:0/4275522081 shutdown_connections 2024-03-25T18:18:45.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.828+0000 7f87291af640 1 -- 172.21.15.122:0/4275522081 wait complete. 2024-03-25T18:18:45.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.829+0000 7f87291af640 10 monclient: build_initial_monmap 2024-03-25T18:18:45.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.829+0000 7f87291af640 10 monclient: monmap: 2024-03-25T18:18:45.832 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:18:45.832 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:45.832 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:18:45.832 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:18:45.832 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:45.832 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:18:45.832 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:18:45.832 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:18:45.832 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:18:45.832 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:45.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.829+0000 7f87291af640 1 librados: starting msgr at 2024-03-25T18:18:45.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.829+0000 7f87291af640 1 librados: starting objecter 2024-03-25T18:18:45.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.829+0000 7f87291af640 1 Processor -- start 2024-03-25T18:18:45.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.830+0000 7f87291af640 1 -- start start 2024-03-25T18:18:45.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.830+0000 7f87291af640 1 librados: setting wanted keys 2024-03-25T18:18:45.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.830+0000 7f87291af640 1 librados: calling monclient init 2024-03-25T18:18:45.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.830+0000 7f87291af640 10 monclient: init 2024-03-25T18:18:45.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.830+0000 7f87291af640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:45.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.830+0000 7f87291af640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:45.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.830+0000 7f87291af640 10 monclient(hunting): picked mon.noname-a con 0x7f87240b1730 addr v1:172.21.15.122:6789/0 2024-03-25T18:18:45.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.830+0000 7f87291af640 10 monclient(hunting): picked mon.noname-c con 0x7f87240b4d60 addr v1:172.21.15.122:6790/0 2024-03-25T18:18:45.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.830+0000 7f87291af640 10 monclient(hunting): picked mon.noname-b con 0x7f872414be90 addr v1:172.21.15.204:6789/0 2024-03-25T18:18:45.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.830+0000 7f87291af640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87241420d0 con 0x7f87240b1730 2024-03-25T18:18:45.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.830+0000 7f87291af640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8724138270 con 0x7f872414be90 2024-03-25T18:18:45.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.830+0000 7f87291af640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8724138400 con 0x7f87240b4d60 2024-03-25T18:18:45.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.830+0000 7f87291af640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:45.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.830+0000 7f87291af640 10 monclient(hunting): authenticate will time out at 7449.096775s 2024-03-25T18:18:45.844 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:02:39.587 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:45.854 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.852+0000 7f871bfff640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f87240b4d60 0x7f87241405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:45820/0 (socket says 172.21.15.122:45820) 2024-03-25T18:18:45.854 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.852+0000 7f871bfff640 1 -- 172.21.15.122:0/498010308 learned_addr learned my addr 172.21.15.122:0/498010308 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:18:45.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.858+0000 7f871b7fe640 10 client.?.objecter ms_handle_connect 0x7f87240b4d60 2024-03-25T18:18:45.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.858+0000 7f871b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:45.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.863+0000 7f871b7fe640 10 client.?.objecter ms_handle_connect 0x7f87240b1730 2024-03-25T18:18:45.865 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.864+0000 7f871b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:45.869 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.868+0000 7f871b7fe640 10 client.?.objecter ms_handle_connect 0x7f872414be90 2024-03-25T18:18:45.869 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.868+0000 7f871b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:45.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.888+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 110971960 0 0) 0x7f8724138270 con 0x7f872414be90 2024-03-25T18:18:45.890 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.888+0000 7f871b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:45.890 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.888+0000 7f871b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:45.890 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.888+0000 7f871b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:45.890 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.888+0000 7f871b7fe640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:18:45.890 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.889+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8704004eb0 con 0x7f872414be90 2024-03-25T18:18:45.898 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.897+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673899497 0 0) 0x7f87241420d0 con 0x7f87240b1730 2024-03-25T18:18:45.898 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.897+0000 7f871b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:45.898 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.897+0000 7f871b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:45.899 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.897+0000 7f871b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:45.899 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.897+0000 7f871b7fe640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:18:45.899 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.897+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8724138270 con 0x7f87240b1730 2024-03-25T18:18:45.910 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.909+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3965384576 0 0) 0x7f8724138400 con 0x7f87240b4d60 2024-03-25T18:18:45.910 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.909+0000 7f871b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:45.910 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.909+0000 7f871b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:45.910 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.909+0000 7f871b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:45.911 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.909+0000 7f871b7fe640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:18:45.911 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.909+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87241420d0 con 0x7f87240b4d60 2024-03-25T18:18:45.930 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.929+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 431358939 0 0) 0x7f8724138270 con 0x7f87240b1730 2024-03-25T18:18:45.930 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.929+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8724138400 con 0x7f87240b1730 2024-03-25T18:18:45.930 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.929+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f87140018a0 con 0x7f87240b1730 2024-03-25T18:18:45.930 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.929+0000 7f871b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:45.930 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.929+0000 7f871b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:45.930 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.929+0000 7f871b7fe640 10 monclient(hunting): dump: 2024-03-25T18:18:45.930 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:45.931 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:45.931 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:45.931 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:45.931 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:45.931 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:45.931 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:45.931 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:45.931 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:45.931 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:45.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.929+0000 7f871b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:45.943 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.941+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2491284422 0 0) 0x7f8724138400 con 0x7f87240b1730 2024-03-25T18:18:45.943 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.941+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 >> v1:172.21.15.122:6790/0 conn(0x7f87240b4d60 legacy=0x7f87241405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.943 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.941+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 >> v1:172.21.15.204:6789/0 conn(0x7f872414be90 legacy=0x7f8724134d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:45.943 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.941+0000 7f871b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:45.943 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.941+0000 7f871b7fe640 1 monclient: found mon.a 2024-03-25T18:18:45.943 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.942+0000 7f871b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:45.943 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.942+0000 7f871b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:18:45.943 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.942+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8724142480 con 0x7f87240b1730 2024-03-25T18:18:45.943 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.942+0000 7f871b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:45.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.942+0000 7f871b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:45.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.942+0000 7f871b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:45.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.942+0000 7f87291af640 5 monclient: authenticate success, global_id 4332 2024-03-25T18:18:45.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.942+0000 7f87291af640 10 monclient: _renew_subs 2024-03-25T18:18:45.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.942+0000 7f87291af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:18:45.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.942+0000 7f87291af640 1 -- 172.21.15.122:0/498010308 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87241426d0 con 0x7f87240b1730 2024-03-25T18:18:45.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.942+0000 7f87291af640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:45.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.942+0000 7f87291af640 10 monclient: _renew_subs 2024-03-25T18:18:45.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.942+0000 7f87291af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:18:45.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.942+0000 7f87291af640 1 -- 172.21.15.122:0/498010308 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87241dba90 con 0x7f87240b1730 2024-03-25T18:18:45.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.942+0000 7f87291af640 1 librados: init done 2024-03-25T18:18:45.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.943+0000 7f87291af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:45.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.943+0000 7f87291af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:45.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.943+0000 7f87291af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:18:45.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:45.943+0000 7f87291af640 1 -- 172.21.15.122:0/498010308 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87241401f0 con 0x7f87240b1730 2024-03-25T18:18:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.252+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8714001cb0 con 0x7f87240b1730 2024-03-25T18:18:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.252+0000 7f871b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.252+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f8714004980 con 0x7f87240b1730 2024-03-25T18:18:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.252+0000 7f871b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.252+0000 7f871b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.252+0000 7f871b7fe640 10 monclient: dump: 2024-03-25T18:18:46.254 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:46.254 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:46.254 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:46.254 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:46.254 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:46.254 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:46.254 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:46.254 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:46.254 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:46.254 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:46.254 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.252+0000 7f872159a640 4 set_mon_vals no callback set 2024-03-25T18:18:46.255 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.252+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f8714004b60 con 0x7f87240b1730 2024-03-25T18:18:46.255 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.253+0000 7f871bfff640 10 monclient: get_auth_request con 0x7f870406de20 auth_method 0 2024-03-25T18:18:46.255 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.253+0000 7f871b7fe640 10 client.4332.objecter ms_handle_connect 0x7f870406de20 2024-03-25T18:18:46.262 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.261+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f87140546e0 con 0x7f87240b1730 2024-03-25T18:18:46.262 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.261+0000 7f871b7fe640 10 client.4332.objecter ms_dispatch 0x7f8724000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:46.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.261+0000 7f871b7fe640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:46.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.261+0000 7f871b7fe640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:46.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.261+0000 7f871b7fe640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-25T18:18:46.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.261+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8714050020 con 0x7f87240b1730 2024-03-25T18:18:46.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.261+0000 7f871b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:46.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.261+0000 7f871b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:46.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.513+0000 7f87291af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:46.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.513+0000 7f87291af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:46.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.513+0000 7f87291af640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:18:46.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.513+0000 7f87291af640 1 -- 172.21.15.122:0/498010308 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f87241daec0 con 0x7f87240b1730 2024-03-25T18:18:46.535 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.534+0000 7f871b7fe640 1 -- 172.21.15.122:0/498010308 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f87241daec0 con 0x7f87240b1730 2024-03-25T18:18:46.535 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.534+0000 7f871b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:46.535 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.534+0000 7f871b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:46.536 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:18:46.536 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:46.538 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.536+0000 7f87291af640 10 librados: watch_flush enter 2024-03-25T18:18:46.538 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.536+0000 7f87291af640 10 librados: watch_flush exit 2024-03-25T18:18:46.538 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.536+0000 7f87291af640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-25T18:18:46.538 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.536+0000 7f87291af640 10 client.4332.objecter successfully canceled tick 2024-03-25T18:18:46.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.536+0000 7f87291af640 1 -- 172.21.15.122:0/498010308 >> v1:172.21.15.204:6800/676228005 conn(0x7f870406de20 legacy=0x7f87040702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:46.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.536+0000 7f87291af640 10 monclient: shutdown 2024-03-25T18:18:46.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.536+0000 7f87291af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:46.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.536+0000 7f87291af640 1 -- 172.21.15.122:0/498010308 >> v1:172.21.15.122:6789/0 conn(0x7f87240b1730 legacy=0x7f87241db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:46.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.537+0000 7f87291af640 1 -- 172.21.15.122:0/498010308 shutdown_connections 2024-03-25T18:18:46.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.537+0000 7f87291af640 1 -- 172.21.15.122:0/498010308 >> 172.21.15.122:0/498010308 conn(0x7f87240617a0 msgr2=0x7f87240b2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:46.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.537+0000 7f87291af640 1 -- 172.21.15.122:0/498010308 shutdown_connections 2024-03-25T18:18:46.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.537+0000 7f87291af640 1 -- 172.21.15.122:0/498010308 wait complete. 2024-03-25T18:18:46.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:46.537+0000 7f87291af640 1 librados: shutdown 2024-03-25T18:18:46.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:53.356 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:53.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.469+0000 7f0f67cff640 10 monclient: get_monmap_and_config 2024-03-25T18:18:53.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.469+0000 7f0f67cff640 10 monclient: build_initial_monmap 2024-03-25T18:18:53.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.469+0000 7f0f67cff640 10 monclient: monmap: 2024-03-25T18:18:53.470 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:18:53.470 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:53.470 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:18:53.470665+0000 2024-03-25T18:18:53.471 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:18:53.470665+0000 2024-03-25T18:18:53.471 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:53.471 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:53.471 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:18:53.471 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:18:53.471 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:18:53.471 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:53.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.470+0000 7f0f67cff640 1 Processor -- start 2024-03-25T18:18:53.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.470+0000 7f0f67cff640 1 -- start start 2024-03-25T18:18:53.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.470+0000 7f0f67cff640 10 monclient: init 2024-03-25T18:18:53.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.471+0000 7f0f67cff640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:53.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.471+0000 7f0f67cff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:53.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.471+0000 7f0f67cff640 10 monclient(hunting): picked mon.noname-b con 0x7f0f600a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:18:53.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.471+0000 7f0f67cff640 10 monclient(hunting): picked mon.noname-a con 0x7f0f6013f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:18:53.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.471+0000 7f0f67cff640 10 monclient(hunting): picked mon.noname-c con 0x7f0f601430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:18:53.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.471+0000 7f0f67cff640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f60142520 con 0x7f0f6013f710 2024-03-25T18:18:53.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.471+0000 7f0f67cff640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f600b28a0 con 0x7f0f601430c0 2024-03-25T18:18:53.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.471+0000 7f0f67cff640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f600b2a30 con 0x7f0f600a98f0 2024-03-25T18:18:53.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.471+0000 7f0f67cff640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:53.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.471+0000 7f0f67cff640 10 monclient(hunting): authenticate will time out at 7456.737740s 2024-03-25T18:18:53.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.476+0000 7f0f6529a640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0f6013f710 0x7f0f601429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:42254/0 (socket says 172.21.15.122:42254) 2024-03-25T18:18:53.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.476+0000 7f0f6529a640 1 -- 172.21.15.122:0/557669053 learned_addr learned my addr 172.21.15.122:0/557669053 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:18:53.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.487+0000 7f0f64a99640 1 -- 172.21.15.122:0/557669053 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 637355090 0 0) 0x7f0f60142520 con 0x7f0f6013f710 2024-03-25T18:18:53.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.487+0000 7f0f64a99640 10 monclient(hunting): _negotiate 2024-03-25T18:18:53.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.487+0000 7f0f64a99640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:53.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.487+0000 7f0f64a99640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:53.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.487+0000 7f0f64a99640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:18:53.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.487+0000 7f0f64a99640 1 -- 172.21.15.122:0/557669053 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f4c003520 con 0x7f0f6013f710 2024-03-25T18:18:53.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.491+0000 7f0f64a99640 1 -- 172.21.15.122:0/557669053 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3471921651 0 0) 0x7f0f600b2a30 con 0x7f0f600a98f0 2024-03-25T18:18:53.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.492+0000 7f0f64a99640 10 monclient(hunting): _negotiate 2024-03-25T18:18:53.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.492+0000 7f0f64a99640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:53.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.492+0000 7f0f64a99640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:53.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.492+0000 7f0f64a99640 10 monclient(hunting): my global_id is 4316 2024-03-25T18:18:53.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.492+0000 7f0f64a99640 1 -- 172.21.15.122:0/557669053 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f60142520 con 0x7f0f600a98f0 2024-03-25T18:18:53.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f64a99640 1 -- 172.21.15.122:0/557669053 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 116264187 0 0) 0x7f0f4c003520 con 0x7f0f6013f710 2024-03-25T18:18:53.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f64a99640 1 -- 172.21.15.122:0/557669053 >> v1:172.21.15.122:6790/0 conn(0x7f0f600a98f0 legacy=0x7f0f600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f64a99640 1 -- 172.21.15.122:0/557669053 >> v1:172.21.15.204:6789/0 conn(0x7f0f601430c0 legacy=0x7f0f6014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f64a99640 10 monclient: _finish_hunting 0 2024-03-25T18:18:53.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f64a99640 1 monclient: found mon.noname-a 2024-03-25T18:18:53.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f64a99640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:53.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f64a99640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:18:53.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f64a99640 1 -- 172.21.15.122:0/557669053 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f600b44c0 con 0x7f0f6013f710 2024-03-25T18:18:53.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f64a99640 10 monclient: _finish_auth 0 2024-03-25T18:18:53.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f64a99640 10 monclient: _check_auth_tickets 2024-03-25T18:18:53.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f64a99640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:53.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f67cff640 5 monclient: authenticate success, global_id 4335 2024-03-25T18:18:53.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f67cff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:53.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f64a99640 1 -- 172.21.15.122:0/557669053 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0f50001690 con 0x7f0f6013f710 2024-03-25T18:18:53.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f64a99640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:53.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f64a99640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:53.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f64a99640 10 monclient: dump: 2024-03-25T18:18:53.497 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:53.497 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:53.497 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:53.498 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:53.498 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:53.498 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:53.498 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:53.498 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:53.498 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:53.498 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:53.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.495+0000 7f0f67cff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:53.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.506+0000 7f0f64a99640 1 -- 172.21.15.122:0/557669053 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f50001870 con 0x7f0f6013f710 2024-03-25T18:18:53.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.506+0000 7f0f64a99640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:53.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.506+0000 7f0f64a99640 1 -- 172.21.15.122:0/557669053 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0f50001690 con 0x7f0f6013f710 2024-03-25T18:18:53.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.506+0000 7f0f64a99640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:53.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.506+0000 7f0f64a99640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:53.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.506+0000 7f0f64a99640 10 monclient: dump: 2024-03-25T18:18:53.508 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:53.508 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:53.508 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:53.508 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:53.508 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:53.508 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:53.508 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:53.508 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:53.508 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:53.508 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:53.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.506+0000 7f0f67cff640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:53.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.506+0000 7f0f67cff640 4 set_mon_vals no callback set 2024-03-25T18:18:53.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.507+0000 7f0f67cff640 10 monclient: shutdown 2024-03-25T18:18:53.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.507+0000 7f0f67cff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:53.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.507+0000 7f0f67cff640 1 -- 172.21.15.122:0/557669053 >> v1:172.21.15.122:6789/0 conn(0x7f0f6013f710 legacy=0x7f0f601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.507+0000 7f0f67cff640 1 -- 172.21.15.122:0/557669053 shutdown_connections 2024-03-25T18:18:53.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.507+0000 7f0f67cff640 1 -- 172.21.15.122:0/557669053 >> 172.21.15.122:0/557669053 conn(0x7f0f600b7ea0 msgr2=0x7f0f600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:53.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.507+0000 7f0f67cff640 1 -- 172.21.15.122:0/557669053 shutdown_connections 2024-03-25T18:18:53.510 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.508+0000 7f0f67cff640 1 -- 172.21.15.122:0/557669053 wait complete. 2024-03-25T18:18:53.510 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.508+0000 7f0f67cff640 10 monclient: build_initial_monmap 2024-03-25T18:18:53.510 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.509+0000 7f0f67cff640 10 monclient: monmap: 2024-03-25T18:18:53.510 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:18:53.511 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:53.511 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:18:53.511 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:18:53.511 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:53.511 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:18:53.511 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:18:53.511 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:18:53.511 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:18:53.511 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:53.511 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.509+0000 7f0f67cff640 1 librados: starting msgr at 2024-03-25T18:18:53.511 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.509+0000 7f0f67cff640 1 librados: starting objecter 2024-03-25T18:18:53.511 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.509+0000 7f0f67cff640 1 Processor -- start 2024-03-25T18:18:53.511 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.509+0000 7f0f67cff640 1 -- start start 2024-03-25T18:18:53.511 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.509+0000 7f0f67cff640 1 librados: setting wanted keys 2024-03-25T18:18:53.511 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.509+0000 7f0f67cff640 1 librados: calling monclient init 2024-03-25T18:18:53.511 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.509+0000 7f0f67cff640 10 monclient: init 2024-03-25T18:18:53.512 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.510+0000 7f0f67cff640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:53.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.510+0000 7f0f67cff640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:53.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.510+0000 7f0f67cff640 10 monclient(hunting): picked mon.noname-c con 0x7f0f600a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:18:53.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.510+0000 7f0f67cff640 10 monclient(hunting): picked mon.noname-b con 0x7f0f6013f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:18:53.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.510+0000 7f0f67cff640 10 monclient(hunting): picked mon.noname-a con 0x7f0f601430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:18:53.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.510+0000 7f0f67cff640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f601390e0 con 0x7f0f601430c0 2024-03-25T18:18:53.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.510+0000 7f0f67cff640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f601db0a0 con 0x7f0f6013f710 2024-03-25T18:18:53.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.510+0000 7f0f67cff640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f60137580 con 0x7f0f600a98f0 2024-03-25T18:18:53.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.510+0000 7f0f67cff640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:53.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.510+0000 7f0f67cff640 10 monclient(hunting): authenticate will time out at 7456.776821s 2024-03-25T18:18:53.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.522+0000 7f0f6529a640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f0f6013f710 0x7f0f601369a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:42836/0 (socket says 172.21.15.122:42836) 2024-03-25T18:18:53.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.522+0000 7f0f6529a640 1 -- 172.21.15.122:0/4158885837 learned_addr learned my addr 172.21.15.122:0/4158885837 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:18:53.524 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.523+0000 7f0f567fc640 10 client.?.objecter ms_handle_connect 0x7f0f6013f710 2024-03-25T18:18:53.524 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.523+0000 7f0f567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:53.534 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.533+0000 7f0f567fc640 1 -- 172.21.15.122:0/4158885837 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3571939896 0 0) 0x7f0f601db0a0 con 0x7f0f6013f710 2024-03-25T18:18:53.534 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.533+0000 7f0f567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:53.534 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.533+0000 7f0f567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:53.534 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.533+0000 7f0f567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:53.534 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.533+0000 7f0f567fc640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:18:53.535 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.533+0000 7f0f567fc640 1 -- 172.21.15.122:0/4158885837 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f44004eb0 con 0x7f0f6013f710 2024-03-25T18:18:53.535 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.533+0000 7f0f567fc640 10 client.?.objecter ms_handle_connect 0x7f0f601430c0 2024-03-25T18:18:53.535 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.533+0000 7f0f567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:53.540 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.539+0000 7f0f567fc640 10 client.?.objecter ms_handle_connect 0x7f0f600a98f0 2024-03-25T18:18:53.540 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.539+0000 7f0f567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:53.544 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.543+0000 7f0f567fc640 1 -- 172.21.15.122:0/4158885837 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3828832424 0 0) 0x7f0f44004eb0 con 0x7f0f6013f710 2024-03-25T18:18:53.544 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.543+0000 7f0f567fc640 1 -- 172.21.15.122:0/4158885837 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f601db0a0 con 0x7f0f6013f710 2024-03-25T18:18:53.544 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.543+0000 7f0f567fc640 1 -- 172.21.15.122:0/4158885837 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0f500047e0 con 0x7f0f6013f710 2024-03-25T18:18:53.544 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.543+0000 7f0f567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:53.544 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.543+0000 7f0f567fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:53.544 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.543+0000 7f0f567fc640 10 monclient(hunting): dump: 2024-03-25T18:18:53.544 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:53.545 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:53.545 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:53.545 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:53.545 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:53.545 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:53.545 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:53.545 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:53.545 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:53.545 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:53.545 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.543+0000 7f0f567fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:53.555 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f567fc640 1 -- 172.21.15.122:0/4158885837 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 924709121 0 0) 0x7f0f601db0a0 con 0x7f0f6013f710 2024-03-25T18:18:53.555 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f567fc640 1 -- 172.21.15.122:0/4158885837 >> v1:172.21.15.122:6790/0 conn(0x7f0f600a98f0 legacy=0x7f0f601da990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.555 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f567fc640 1 -- 172.21.15.122:0/4158885837 >> v1:172.21.15.122:6789/0 conn(0x7f0f601430c0 legacy=0x7f0f60133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.555 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f567fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:53.555 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f567fc640 1 monclient: found mon.b 2024-03-25T18:18:53.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:53.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f567fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:53.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f567fc640 1 -- 172.21.15.122:0/4158885837 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f601370b0 con 0x7f0f6013f710 2024-03-25T18:18:53.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f567fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:53.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f567fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:53.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:53.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f67cff640 5 monclient: authenticate success, global_id 4336 2024-03-25T18:18:53.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f67cff640 10 monclient: _renew_subs 2024-03-25T18:18:53.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f67cff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:53.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f67cff640 1 -- 172.21.15.122:0/4158885837 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f601337a0 con 0x7f0f6013f710 2024-03-25T18:18:53.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f67cff640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:53.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f67cff640 10 monclient: _renew_subs 2024-03-25T18:18:53.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f67cff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:53.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f67cff640 1 -- 172.21.15.122:0/4158885837 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f600b7350 con 0x7f0f6013f710 2024-03-25T18:18:53.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.554+0000 7f0f67cff640 1 librados: init done 2024-03-25T18:18:53.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.555+0000 7f0f67cff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:53.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.555+0000 7f0f67cff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:53.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.555+0000 7f0f67cff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:53.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.555+0000 7f0f67cff640 1 -- 172.21.15.122:0/4158885837 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f60133af0 con 0x7f0f6013f710 2024-03-25T18:18:53.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.833+0000 7f0f567fc640 1 -- 172.21.15.122:0/4158885837 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f50004bf0 con 0x7f0f6013f710 2024-03-25T18:18:53.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.833+0000 7f0f567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:53.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.833+0000 7f0f567fc640 1 -- 172.21.15.122:0/4158885837 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0f50004f00 con 0x7f0f6013f710 2024-03-25T18:18:53.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.833+0000 7f0f567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:53.835 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.833+0000 7f0f567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:53.835 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.833+0000 7f0f567fc640 10 monclient: dump: 2024-03-25T18:18:53.835 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:18:53.835 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:18:53.835 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:53.835 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:18:53.835 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:53.835 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:18:53.835 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:18:53.835 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:18:53.835 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:18:53.835 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:18:53.835 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.833+0000 7f0f57fff640 4 set_mon_vals no callback set 2024-03-25T18:18:53.836 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.834+0000 7f0f567fc640 1 -- 172.21.15.122:0/4158885837 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f0f5001aed0 con 0x7f0f6013f710 2024-03-25T18:18:53.836 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.835+0000 7f0f65a9b640 10 monclient: get_auth_request con 0x7f0f4406dc50 auth_method 0 2024-03-25T18:18:53.836 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.835+0000 7f0f567fc640 10 client.4336.objecter ms_handle_connect 0x7f0f4406dc50 2024-03-25T18:18:53.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.840+0000 7f0f567fc640 1 -- 172.21.15.122:0/4158885837 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f0f50054a50 con 0x7f0f6013f710 2024-03-25T18:18:53.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.840+0000 7f0f567fc640 10 client.4336.objecter ms_dispatch 0x7f0f60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:53.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.840+0000 7f0f567fc640 3 client.4336.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:53.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.840+0000 7f0f567fc640 3 client.4336.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:53.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.840+0000 7f0f567fc640 20 client.4336.objecter dump_active .. 0 homeless 2024-03-25T18:18:53.842 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.840+0000 7f0f567fc640 1 -- 172.21.15.122:0/4158885837 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f50050390 con 0x7f0f6013f710 2024-03-25T18:18:53.842 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.840+0000 7f0f567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:53.842 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:53.840+0000 7f0f567fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:54.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.088+0000 7f0f67cff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:54.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.088+0000 7f0f67cff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:54.090 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.088+0000 7f0f67cff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:18:54.090 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.088+0000 7f0f67cff640 1 -- 172.21.15.122:0/4158885837 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f600627b0 con 0x7f0f6013f710 2024-03-25T18:18:54.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.121+0000 7f0f567fc640 1 -- 172.21.15.122:0/4158885837 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f0f50052490 con 0x7f0f6013f710 2024-03-25T18:18:54.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.121+0000 7f0f567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:54.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.121+0000 7f0f567fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:54.122 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:18:54.122 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:54.125 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.123+0000 7f0f67cff640 10 librados: watch_flush enter 2024-03-25T18:18:54.125 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.123+0000 7f0f67cff640 10 librados: watch_flush exit 2024-03-25T18:18:54.125 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.123+0000 7f0f67cff640 20 client.4336.objecter shutdown clearing up homeless session... 2024-03-25T18:18:54.125 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.123+0000 7f0f67cff640 10 client.4336.objecter successfully canceled tick 2024-03-25T18:18:54.125 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.123+0000 7f0f67cff640 1 -- 172.21.15.122:0/4158885837 >> v1:172.21.15.204:6800/676228005 conn(0x7f0f4406dc50 legacy=0x7f0f44070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:54.125 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.123+0000 7f0f67cff640 10 monclient: shutdown 2024-03-25T18:18:54.125 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.123+0000 7f0f67cff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:54.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.123+0000 7f0f67cff640 1 -- 172.21.15.122:0/4158885837 >> v1:172.21.15.204:6789/0 conn(0x7f0f6013f710 legacy=0x7f0f601369a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:54.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.124+0000 7f0f67cff640 1 -- 172.21.15.122:0/4158885837 shutdown_connections 2024-03-25T18:18:54.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.124+0000 7f0f67cff640 1 -- 172.21.15.122:0/4158885837 >> 172.21.15.122:0/4158885837 conn(0x7f0f600b7ea0 msgr2=0x7f0f6013ca30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:54.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.124+0000 7f0f67cff640 1 -- 172.21.15.122:0/4158885837 shutdown_connections 2024-03-25T18:18:54.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.124+0000 7f0f67cff640 1 -- 172.21.15.122:0/4158885837 wait complete. 2024-03-25T18:18:54.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:18:54.124+0000 7f0f67cff640 1 librados: shutdown 2024-03-25T18:18:54.136 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:00.943 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:01.058 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.057+0000 7f6fff5c6640 10 monclient: get_monmap_and_config 2024-03-25T18:19:01.058 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.057+0000 7f6fff5c6640 10 monclient: build_initial_monmap 2024-03-25T18:19:01.058 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.057+0000 7f6fff5c6640 10 monclient: monmap: 2024-03-25T18:19:01.058 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:19:01.058 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:01.058 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:19:01.058503+0000 2024-03-25T18:19:01.059 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:19:01.058503+0000 2024-03-25T18:19:01.059 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:01.059 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:01.059 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:19:01.059 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:19:01.059 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:19:01.059 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:01.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.058+0000 7f6fff5c6640 1 Processor -- start 2024-03-25T18:19:01.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.058+0000 7f6fff5c6640 1 -- start start 2024-03-25T18:19:01.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.058+0000 7f6fff5c6640 10 monclient: init 2024-03-25T18:19:01.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.059+0000 7f6fff5c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:01.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.059+0000 7f6fff5c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:01.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.059+0000 7f6fff5c6640 10 monclient(hunting): picked mon.noname-a con 0x7f6ff80b4d60 addr v1:172.21.15.122:6789/0 2024-03-25T18:19:01.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.059+0000 7f6fff5c6640 10 monclient(hunting): picked mon.noname-b con 0x7f6ff80b1730 addr v1:172.21.15.122:6790/0 2024-03-25T18:19:01.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.059+0000 7f6fff5c6640 10 monclient(hunting): picked mon.noname-c con 0x7f6ff814be90 addr v1:172.21.15.204:6789/0 2024-03-25T18:19:01.061 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.059+0000 7f6fff5c6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff80af940 con 0x7f6ff80b4d60 2024-03-25T18:19:01.061 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.059+0000 7f6fff5c6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff80b0a70 con 0x7f6ff814be90 2024-03-25T18:19:01.061 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.059+0000 7f6fff5c6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff80b0d50 con 0x7f6ff80b1730 2024-03-25T18:19:01.061 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.059+0000 7f6fff5c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:01.061 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.059+0000 7f6fff5c6640 10 monclient(hunting): authenticate will time out at 7464.325323s 2024-03-25T18:19:01.067 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.066+0000 7f6ffcb61640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f6ff80b1730 0x7f6ff80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:33506/0 (socket says 172.21.15.122:33506) 2024-03-25T18:19:01.067 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.066+0000 7f6ffcb61640 1 -- 172.21.15.122:0/1353951996 learned_addr learned my addr 172.21.15.122:0/1353951996 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:19:01.076 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.075+0000 7f6feffff640 1 -- 172.21.15.122:0/1353951996 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624801124 0 0) 0x7f6ff80b0d50 con 0x7f6ff80b1730 2024-03-25T18:19:01.076 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.075+0000 7f6feffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:01.076 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.075+0000 7f6feffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:01.076 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.075+0000 7f6feffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:01.076 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.075+0000 7f6feffff640 10 monclient(hunting): my global_id is 4322 2024-03-25T18:19:01.076 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.075+0000 7f6feffff640 1 -- 172.21.15.122:0/1353951996 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe4003520 con 0x7f6ff80b1730 2024-03-25T18:19:01.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.076+0000 7f6feffff640 1 -- 172.21.15.122:0/1353951996 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914631942 0 0) 0x7f6ff80b0a70 con 0x7f6ff814be90 2024-03-25T18:19:01.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.076+0000 7f6feffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:01.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.076+0000 7f6feffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:01.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.076+0000 7f6feffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:01.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.076+0000 7f6feffff640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:19:01.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.076+0000 7f6feffff640 1 -- 172.21.15.122:0/1353951996 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff80b0d50 con 0x7f6ff814be90 2024-03-25T18:19:01.081 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6feffff640 1 -- 172.21.15.122:0/1353951996 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1902234491 0 0) 0x7f6fe4003520 con 0x7f6ff80b1730 2024-03-25T18:19:01.081 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6feffff640 1 -- 172.21.15.122:0/1353951996 >> v1:172.21.15.204:6789/0 conn(0x7f6ff814be90 legacy=0x7f6ff814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6feffff640 1 -- 172.21.15.122:0/1353951996 >> v1:172.21.15.122:6789/0 conn(0x7f6ff80b4d60 legacy=0x7f6ff813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6feffff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:01.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6feffff640 1 monclient: found mon.noname-b 2024-03-25T18:19:01.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6feffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:01.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6feffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:19:01.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6feffff640 1 -- 172.21.15.122:0/1353951996 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff80b1030 con 0x7f6ff80b1730 2024-03-25T18:19:01.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6feffff640 10 monclient: _finish_auth 0 2024-03-25T18:19:01.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6feffff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:01.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6feffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:01.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6feffff640 1 -- 172.21.15.122:0/1353951996 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f6fe8001690 con 0x7f6ff80b1730 2024-03-25T18:19:01.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6fff5c6640 5 monclient: authenticate success, global_id 4322 2024-03-25T18:19:01.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6fff5c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:01.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6feffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:01.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6feffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:01.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6feffff640 10 monclient: dump: 2024-03-25T18:19:01.083 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:01.083 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:01.083 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:01.083 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:01.083 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:01.083 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:01.083 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:01.083 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:01.083 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:01.083 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:01.083 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.080+0000 7f6fff5c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:01.094 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.093+0000 7f6feffff640 1 -- 172.21.15.122:0/1353951996 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fe80017f0 con 0x7f6ff80b1730 2024-03-25T18:19:01.094 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.093+0000 7f6feffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:01.094 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.093+0000 7f6feffff640 1 -- 172.21.15.122:0/1353951996 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f6fe8001c00 con 0x7f6ff80b1730 2024-03-25T18:19:01.094 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.093+0000 7f6fff5c6640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:01.095 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.093+0000 7f6fff5c6640 4 set_mon_vals no callback set 2024-03-25T18:19:01.095 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.093+0000 7f6feffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:01.095 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.093+0000 7f6feffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:01.095 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.093+0000 7f6feffff640 10 monclient: dump: 2024-03-25T18:19:01.095 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:01.095 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:01.095 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:01.095 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:01.095 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:01.095 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:01.095 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:01.095 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:01.095 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:01.095 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:01.096 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.094+0000 7f6fff5c6640 10 monclient: shutdown 2024-03-25T18:19:01.096 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.094+0000 7f6fff5c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:01.096 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.094+0000 7f6fff5c6640 1 -- 172.21.15.122:0/1353951996 >> v1:172.21.15.122:6790/0 conn(0x7f6ff80b1730 legacy=0x7f6ff80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.096 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.094+0000 7f6fff5c6640 1 -- 172.21.15.122:0/1353951996 shutdown_connections 2024-03-25T18:19:01.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.094+0000 7f6fff5c6640 1 -- 172.21.15.122:0/1353951996 >> 172.21.15.122:0/1353951996 conn(0x7f6ff80617a0 msgr2=0x7f6ff8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:01.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.094+0000 7f6fff5c6640 1 -- 172.21.15.122:0/1353951996 shutdown_connections 2024-03-25T18:19:01.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.094+0000 7f6fff5c6640 1 -- 172.21.15.122:0/1353951996 wait complete. 2024-03-25T18:19:01.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.095+0000 7f6fff5c6640 10 monclient: build_initial_monmap 2024-03-25T18:19:01.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.095+0000 7f6fff5c6640 10 monclient: monmap: 2024-03-25T18:19:01.097 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:19:01.097 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:01.097 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:19:01.097 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:19:01.097 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:01.097 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:19:01.097 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:19:01.097 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:19:01.097 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:19:01.097 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:01.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.096+0000 7f6fff5c6640 1 librados: starting msgr at 2024-03-25T18:19:01.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.096+0000 7f6fff5c6640 1 librados: starting objecter 2024-03-25T18:19:01.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.096+0000 7f6fff5c6640 1 Processor -- start 2024-03-25T18:19:01.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.096+0000 7f6fff5c6640 1 -- start start 2024-03-25T18:19:01.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.096+0000 7f6fff5c6640 1 librados: setting wanted keys 2024-03-25T18:19:01.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.096+0000 7f6fff5c6640 1 librados: calling monclient init 2024-03-25T18:19:01.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.096+0000 7f6fff5c6640 10 monclient: init 2024-03-25T18:19:01.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.096+0000 7f6fff5c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:01.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.097+0000 7f6fff5c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:01.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.097+0000 7f6fff5c6640 10 monclient(hunting): picked mon.noname-a con 0x7f6ff80b1730 addr v1:172.21.15.122:6789/0 2024-03-25T18:19:01.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.097+0000 7f6fff5c6640 10 monclient(hunting): picked mon.noname-b con 0x7f6ff80b4d60 addr v1:172.21.15.204:6789/0 2024-03-25T18:19:01.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.097+0000 7f6fff5c6640 10 monclient(hunting): picked mon.noname-c con 0x7f6ff814be90 addr v1:172.21.15.122:6790/0 2024-03-25T18:19:01.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.097+0000 7f6fff5c6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff8140380 con 0x7f6ff80b1730 2024-03-25T18:19:01.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.097+0000 7f6fff5c6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff806fa60 con 0x7f6ff80b4d60 2024-03-25T18:19:01.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.097+0000 7f6fff5c6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff806fd40 con 0x7f6ff814be90 2024-03-25T18:19:01.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.097+0000 7f6fff5c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:01.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.097+0000 7f6fff5c6640 10 monclient(hunting): authenticate will time out at 7464.363062s 2024-03-25T18:19:01.111 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.110+0000 7f6ffcb61640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f6ff80b4d60 0x7f6ff8138810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:36880/0 (socket says 172.21.15.122:36880) 2024-03-25T18:19:01.111 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.110+0000 7f6ffcb61640 1 -- 172.21.15.122:0/4047901638 learned_addr learned my addr 172.21.15.122:0/4047901638 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:19:01.112 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.111+0000 7f6fedffb640 10 client.?.objecter ms_handle_connect 0x7f6ff80b4d60 2024-03-25T18:19:01.112 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.111+0000 7f6fedffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:01.113 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.112+0000 7f6fedffb640 10 client.?.objecter ms_handle_connect 0x7f6ff814be90 2024-03-25T18:19:01.114 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.112+0000 7f6fedffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:01.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.120+0000 7f6fedffb640 1 -- 172.21.15.122:0/4047901638 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875440027 0 0) 0x7f6ff806fa60 con 0x7f6ff80b4d60 2024-03-25T18:19:01.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.120+0000 7f6fedffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:01.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.120+0000 7f6fedffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:01.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.120+0000 7f6fedffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:01.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.120+0000 7f6fedffb640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:19:01.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.120+0000 7f6fedffb640 1 -- 172.21.15.122:0/4047901638 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fdc004eb0 con 0x7f6ff80b4d60 2024-03-25T18:19:01.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.130+0000 7f6fedffb640 1 -- 172.21.15.122:0/4047901638 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2883160726 0 0) 0x7f6fdc004eb0 con 0x7f6ff80b4d60 2024-03-25T18:19:01.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.130+0000 7f6fedffb640 1 -- 172.21.15.122:0/4047901638 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ff806fa60 con 0x7f6ff80b4d60 2024-03-25T18:19:01.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.130+0000 7f6fedffb640 1 -- 172.21.15.122:0/4047901638 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f6fe8001bc0 con 0x7f6ff80b4d60 2024-03-25T18:19:01.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.130+0000 7f6fedffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:01.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.130+0000 7f6fedffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:01.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.130+0000 7f6fedffb640 10 monclient(hunting): dump: 2024-03-25T18:19:01.131 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:01.131 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:01.132 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:01.132 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:01.132 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:01.132 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:01.132 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:01.132 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:01.132 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:01.132 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:01.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.130+0000 7f6fedffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:01.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.132+0000 7f6fedffb640 10 client.?.objecter ms_handle_connect 0x7f6ff80b1730 2024-03-25T18:19:01.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.132+0000 7f6fedffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:01.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.137+0000 7f6fedffb640 1 -- 172.21.15.122:0/4047901638 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 619922474 0 0) 0x7f6ff806fa60 con 0x7f6ff80b4d60 2024-03-25T18:19:01.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.137+0000 7f6fedffb640 1 -- 172.21.15.122:0/4047901638 >> v1:172.21.15.122:6790/0 conn(0x7f6ff814be90 legacy=0x7f6ff81dbcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.137+0000 7f6fedffb640 1 -- 172.21.15.122:0/4047901638 >> v1:172.21.15.122:6789/0 conn(0x7f6ff80b1730 legacy=0x7f6ff806e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.137+0000 7f6fedffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:01.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.137+0000 7f6fedffb640 1 monclient: found mon.b 2024-03-25T18:19:01.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.137+0000 7f6fedffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:01.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.137+0000 7f6fedffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:19:01.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.137+0000 7f6fedffb640 1 -- 172.21.15.122:0/4047901638 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff806e800 con 0x7f6ff80b4d60 2024-03-25T18:19:01.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.137+0000 7f6fedffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:01.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.137+0000 7f6fedffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:01.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.137+0000 7f6fedffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:01.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.137+0000 7f6fff5c6640 5 monclient: authenticate success, global_id 4342 2024-03-25T18:19:01.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.137+0000 7f6fff5c6640 10 monclient: _renew_subs 2024-03-25T18:19:01.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.137+0000 7f6fff5c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:19:01.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.137+0000 7f6fff5c6640 1 -- 172.21.15.122:0/4047901638 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff81d94a0 con 0x7f6ff80b4d60 2024-03-25T18:19:01.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.138+0000 7f6fff5c6640 10 client.4342.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:01.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.138+0000 7f6fff5c6640 10 monclient: _renew_subs 2024-03-25T18:19:01.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.138+0000 7f6fff5c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:19:01.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.138+0000 7f6fff5c6640 1 -- 172.21.15.122:0/4047901638 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff8139100 con 0x7f6ff80b4d60 2024-03-25T18:19:01.140 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.138+0000 7f6fff5c6640 1 librados: init done 2024-03-25T18:19:01.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.139+0000 7f6fff5c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:01.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.139+0000 7f6fff5c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:01.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.139+0000 7f6fff5c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:19:01.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.139+0000 7f6fff5c6640 1 -- 172.21.15.122:0/4047901638 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff806f830 con 0x7f6ff80b4d60 2024-03-25T18:19:01.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.406+0000 7f6fedffb640 1 -- 172.21.15.122:0/4047901638 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fe8004a90 con 0x7f6ff80b4d60 2024-03-25T18:19:01.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.406+0000 7f6fedffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:01.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.406+0000 7f6fedffb640 1 -- 172.21.15.122:0/4047901638 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f6fe8001250 con 0x7f6ff80b4d60 2024-03-25T18:19:01.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.406+0000 7f6fedffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:01.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.407+0000 7f6fedffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:01.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.407+0000 7f6fedffb640 10 monclient: dump: 2024-03-25T18:19:01.408 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:01.408 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:01.408 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:01.408 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:01.408 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:01.409 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:01.409 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:01.409 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:01.409 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:01.409 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:01.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.407+0000 7f6fef7fe640 4 set_mon_vals no callback set 2024-03-25T18:19:01.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.407+0000 7f6fedffb640 1 -- 172.21.15.122:0/4047901638 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f6fe8004fa0 con 0x7f6ff80b4d60 2024-03-25T18:19:01.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.408+0000 7f6ffd362640 10 monclient: get_auth_request con 0x7f6fdc06dc70 auth_method 0 2024-03-25T18:19:01.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.409+0000 7f6fedffb640 10 client.4342.objecter ms_handle_connect 0x7f6fdc06dc70 2024-03-25T18:19:01.413 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.412+0000 7f6fedffb640 1 -- 172.21.15.122:0/4047901638 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f6fe8054950 con 0x7f6ff80b4d60 2024-03-25T18:19:01.413 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.412+0000 7f6fedffb640 10 client.4342.objecter ms_dispatch 0x7f6ff8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:01.413 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.412+0000 7f6fedffb640 3 client.4342.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:01.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.412+0000 7f6fedffb640 3 client.4342.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:01.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.412+0000 7f6fedffb640 20 client.4342.objecter dump_active .. 0 homeless 2024-03-25T18:19:01.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.414+0000 7f6fedffb640 1 -- 172.21.15.122:0/4047901638 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6fe8050460 con 0x7f6ff80b4d60 2024-03-25T18:19:01.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.414+0000 7f6fedffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:01.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.414+0000 7f6fedffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:01.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.671+0000 7f6fff5c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:01.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.671+0000 7f6fff5c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:01.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.671+0000 7f6fff5c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:19:01.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.671+0000 7f6fff5c6640 1 -- 172.21.15.122:0/4047901638 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6ff80627b0 con 0x7f6ff80b4d60 2024-03-25T18:19:01.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.703+0000 7f6fedffb640 1 -- 172.21.15.122:0/4047901638 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f6fe8052390 con 0x7f6ff80b4d60 2024-03-25T18:19:01.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.703+0000 7f6fedffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:01.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.703+0000 7f6fedffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:01.705 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:19:01.705 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:01.708 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.706+0000 7f6fff5c6640 10 librados: watch_flush enter 2024-03-25T18:19:01.708 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.706+0000 7f6fff5c6640 10 librados: watch_flush exit 2024-03-25T18:19:01.708 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.706+0000 7f6fff5c6640 20 client.4342.objecter shutdown clearing up homeless session... 2024-03-25T18:19:01.708 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.706+0000 7f6fff5c6640 10 client.4342.objecter successfully canceled tick 2024-03-25T18:19:01.708 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.706+0000 7f6fff5c6640 1 -- 172.21.15.122:0/4047901638 >> v1:172.21.15.204:6800/676228005 conn(0x7f6fdc06dc70 legacy=0x7f6fdc070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.708 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.706+0000 7f6fff5c6640 10 monclient: shutdown 2024-03-25T18:19:01.708 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.706+0000 7f6fff5c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:01.708 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.706+0000 7f6fff5c6640 1 -- 172.21.15.122:0/4047901638 >> v1:172.21.15.204:6789/0 conn(0x7f6ff80b4d60 legacy=0x7f6ff8138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.708 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.706+0000 7f6fff5c6640 1 -- 172.21.15.122:0/4047901638 shutdown_connections 2024-03-25T18:19:01.708 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.706+0000 7f6fff5c6640 1 -- 172.21.15.122:0/4047901638 >> 172.21.15.122:0/4047901638 conn(0x7f6ff80617a0 msgr2=0x7f6ff813b4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:01.708 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.706+0000 7f6fff5c6640 1 -- 172.21.15.122:0/4047901638 shutdown_connections 2024-03-25T18:19:01.708 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.706+0000 7f6fff5c6640 1 -- 172.21.15.122:0/4047901638 wait complete. 2024-03-25T18:19:01.708 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:01.706+0000 7f6fff5c6640 1 librados: shutdown 2024-03-25T18:19:01.719 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:05.769 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:02:59.512 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:05.773 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:02:59.516 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:05.804 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:02:59.546 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:05.832 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:02:59.578 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:05.834 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:02:59.580 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:05.837 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:02:59.580 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:05.842 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:02:59.585 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:05.865 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:02:59.610 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:05.877 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:02:59.619 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:08.525 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:08.639 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.638+0000 7ff4654e6640 10 monclient: get_monmap_and_config 2024-03-25T18:19:08.639 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.638+0000 7ff4654e6640 10 monclient: build_initial_monmap 2024-03-25T18:19:08.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.638+0000 7ff4654e6640 10 monclient: monmap: 2024-03-25T18:19:08.640 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:19:08.640 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:08.640 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:19:08.639740+0000 2024-03-25T18:19:08.640 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:19:08.639740+0000 2024-03-25T18:19:08.640 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:08.640 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:08.640 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:19:08.640 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:19:08.640 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:19:08.641 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:08.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.639+0000 7ff4654e6640 1 Processor -- start 2024-03-25T18:19:08.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.639+0000 7ff4654e6640 1 -- start start 2024-03-25T18:19:08.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.639+0000 7ff4654e6640 10 monclient: init 2024-03-25T18:19:08.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.639+0000 7ff4654e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:08.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.639+0000 7ff4654e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:08.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.639+0000 7ff4654e6640 10 monclient(hunting): picked mon.noname-c con 0x7ff4600a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:19:08.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.639+0000 7ff4654e6640 10 monclient(hunting): picked mon.noname-a con 0x7ff46013f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:19:08.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.639+0000 7ff4654e6640 10 monclient(hunting): picked mon.noname-b con 0x7ff4601430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:19:08.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.640+0000 7ff4654e6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff460142520 con 0x7ff46013f710 2024-03-25T18:19:08.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.640+0000 7ff4654e6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4600b28a0 con 0x7ff4600a98f0 2024-03-25T18:19:08.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.640+0000 7ff4654e6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4600b2a30 con 0x7ff4601430c0 2024-03-25T18:19:08.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.640+0000 7ff4654e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:08.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.640+0000 7ff4654e6640 10 monclient(hunting): authenticate will time out at 7471.906059s 2024-03-25T18:19:08.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.653+0000 7ff45f7fe640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7ff4601430c0 0x7ff46014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:38118/0 (socket says 172.21.15.122:38118) 2024-03-25T18:19:08.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.653+0000 7ff45f7fe640 1 -- 172.21.15.122:0/3660597243 learned_addr learned my addr 172.21.15.122:0/3660597243 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:19:08.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.665+0000 7ff45dffb640 1 -- 172.21.15.122:0/3660597243 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2574994659 0 0) 0x7ff4600b2a30 con 0x7ff4601430c0 2024-03-25T18:19:08.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.665+0000 7ff45dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.665+0000 7ff45dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.665+0000 7ff45dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.665+0000 7ff45dffb640 10 monclient(hunting): my global_id is 4328 2024-03-25T18:19:08.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.665+0000 7ff45dffb640 1 -- 172.21.15.122:0/3660597243 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff448003520 con 0x7ff4601430c0 2024-03-25T18:19:08.675 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.674+0000 7ff45dffb640 1 -- 172.21.15.122:0/3660597243 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918066473 0 0) 0x7ff4600b28a0 con 0x7ff4600a98f0 2024-03-25T18:19:08.676 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.674+0000 7ff45dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.676 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.674+0000 7ff45dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.676 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.674+0000 7ff45dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.676 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.674+0000 7ff45dffb640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:19:08.676 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.674+0000 7ff45dffb640 1 -- 172.21.15.122:0/3660597243 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4600b2a30 con 0x7ff4600a98f0 2024-03-25T18:19:08.742 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.740+0000 7ff45dffb640 1 -- 172.21.15.122:0/3660597243 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333787112 0 0) 0x7ff460142520 con 0x7ff46013f710 2024-03-25T18:19:08.742 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.740+0000 7ff45dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.742 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.740+0000 7ff45dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.742 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.740+0000 7ff45dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.742 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.740+0000 7ff45dffb640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:19:08.742 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.740+0000 7ff45dffb640 1 -- 172.21.15.122:0/3660597243 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4600b28a0 con 0x7ff46013f710 2024-03-25T18:19:08.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff45dffb640 1 -- 172.21.15.122:0/3660597243 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3997445406 0 0) 0x7ff4600b28a0 con 0x7ff46013f710 2024-03-25T18:19:08.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff45dffb640 1 -- 172.21.15.122:0/3660597243 >> v1:172.21.15.122:6790/0 conn(0x7ff4601430c0 legacy=0x7ff46014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff45dffb640 1 -- 172.21.15.122:0/3660597243 >> v1:172.21.15.204:6789/0 conn(0x7ff4600a98f0 legacy=0x7ff4600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff45dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:08.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff45dffb640 1 monclient: found mon.noname-a 2024-03-25T18:19:08.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff45dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:08.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff45dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:19:08.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff45dffb640 1 -- 172.21.15.122:0/3660597243 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4600b44c0 con 0x7ff46013f710 2024-03-25T18:19:08.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff45dffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:08.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff45dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:08.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff45dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:08.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff45dffb640 1 -- 172.21.15.122:0/3660597243 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff4540015b0 con 0x7ff46013f710 2024-03-25T18:19:08.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff4654e6640 5 monclient: authenticate success, global_id 4347 2024-03-25T18:19:08.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff4654e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:08.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff45dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:08.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff45dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:08.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff45dffb640 10 monclient: dump: 2024-03-25T18:19:08.755 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:08.756 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:08.756 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:08.756 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:08.756 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:08.756 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:08.756 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:08.756 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:08.756 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:08.756 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:08.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.752+0000 7ff4654e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:08.759 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.757+0000 7ff45dffb640 1 -- 172.21.15.122:0/3660597243 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff454001710 con 0x7ff46013f710 2024-03-25T18:19:08.759 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.757+0000 7ff45dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:08.759 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.757+0000 7ff45dffb640 1 -- 172.21.15.122:0/3660597243 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff454001b20 con 0x7ff46013f710 2024-03-25T18:19:08.759 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.757+0000 7ff4654e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:08.759 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.757+0000 7ff4654e6640 4 set_mon_vals no callback set 2024-03-25T18:19:08.759 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.757+0000 7ff45dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:08.759 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.757+0000 7ff45dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:08.759 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.757+0000 7ff45dffb640 10 monclient: dump: 2024-03-25T18:19:08.759 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:08.759 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:08.760 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:08.760 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:08.760 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:08.760 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:08.760 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:08.760 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:08.760 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:08.760 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:08.763 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.758+0000 7ff4654e6640 10 monclient: shutdown 2024-03-25T18:19:08.763 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.758+0000 7ff4654e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:08.763 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.758+0000 7ff4654e6640 1 -- 172.21.15.122:0/3660597243 >> v1:172.21.15.122:6789/0 conn(0x7ff46013f710 legacy=0x7ff4601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.763 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.758+0000 7ff4654e6640 1 -- 172.21.15.122:0/3660597243 shutdown_connections 2024-03-25T18:19:08.763 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.758+0000 7ff4654e6640 1 -- 172.21.15.122:0/3660597243 >> 172.21.15.122:0/3660597243 conn(0x7ff4600b7ea0 msgr2=0x7ff4600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:08.763 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.758+0000 7ff4654e6640 1 -- 172.21.15.122:0/3660597243 shutdown_connections 2024-03-25T18:19:08.764 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.758+0000 7ff4654e6640 1 -- 172.21.15.122:0/3660597243 wait complete. 2024-03-25T18:19:08.764 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.759+0000 7ff4654e6640 10 monclient: build_initial_monmap 2024-03-25T18:19:08.764 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.759+0000 7ff4654e6640 10 monclient: monmap: 2024-03-25T18:19:08.764 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:19:08.764 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:08.764 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:19:08.764 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:19:08.764 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:08.764 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:19:08.764 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:19:08.764 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:19:08.765 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:19:08.765 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:08.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.759+0000 7ff4654e6640 1 librados: starting msgr at 2024-03-25T18:19:08.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.759+0000 7ff4654e6640 1 librados: starting objecter 2024-03-25T18:19:08.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.759+0000 7ff4654e6640 1 Processor -- start 2024-03-25T18:19:08.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.759+0000 7ff4654e6640 1 -- start start 2024-03-25T18:19:08.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.759+0000 7ff4654e6640 1 librados: setting wanted keys 2024-03-25T18:19:08.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.759+0000 7ff4654e6640 1 librados: calling monclient init 2024-03-25T18:19:08.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.759+0000 7ff4654e6640 10 monclient: init 2024-03-25T18:19:08.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.760+0000 7ff4654e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:08.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.760+0000 7ff4654e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:08.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.760+0000 7ff4654e6640 10 monclient(hunting): picked mon.noname-a con 0x7ff4600a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:19:08.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.760+0000 7ff4654e6640 10 monclient(hunting): picked mon.noname-b con 0x7ff46013f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:19:08.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.760+0000 7ff4654e6640 10 monclient(hunting): picked mon.noname-c con 0x7ff4601430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:19:08.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.760+0000 7ff4654e6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4601d7830 con 0x7ff4600a98f0 2024-03-25T18:19:08.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.760+0000 7ff4654e6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4600ab9e0 con 0x7ff46013f710 2024-03-25T18:19:08.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.760+0000 7ff4654e6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4601d7500 con 0x7ff4601430c0 2024-03-25T18:19:08.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.760+0000 7ff4654e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:08.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.760+0000 7ff4654e6640 10 monclient(hunting): authenticate will time out at 7472.026861s 2024-03-25T18:19:08.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.762+0000 7ff45effd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff4600a98f0 0x7ff460138760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:52618/0 (socket says 172.21.15.122:52618) 2024-03-25T18:19:08.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.762+0000 7ff45effd640 1 -- 172.21.15.122:0/4105144349 learned_addr learned my addr 172.21.15.122:0/4105144349 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:19:08.767 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.763+0000 7ff4477fe640 10 client.?.objecter ms_handle_connect 0x7ff4600a98f0 2024-03-25T18:19:08.767 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.763+0000 7ff4477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:08.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.773+0000 7ff4477fe640 1 -- 172.21.15.122:0/4105144349 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991023607 0 0) 0x7ff4601d7830 con 0x7ff4600a98f0 2024-03-25T18:19:08.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.773+0000 7ff4477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.773+0000 7ff4477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.773+0000 7ff4477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.773+0000 7ff4477fe640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:19:08.775 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.773+0000 7ff4477fe640 1 -- 172.21.15.122:0/4105144349 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff44c004eb0 con 0x7ff4600a98f0 2024-03-25T18:19:08.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.788+0000 7ff4477fe640 1 -- 172.21.15.122:0/4105144349 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1260183805 0 0) 0x7ff44c004eb0 con 0x7ff4600a98f0 2024-03-25T18:19:08.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.788+0000 7ff4477fe640 1 -- 172.21.15.122:0/4105144349 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff4601d7830 con 0x7ff4600a98f0 2024-03-25T18:19:08.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.788+0000 7ff4477fe640 1 -- 172.21.15.122:0/4105144349 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff450001a10 con 0x7ff4600a98f0 2024-03-25T18:19:08.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.788+0000 7ff4477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:08.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.788+0000 7ff4477fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:08.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.788+0000 7ff4477fe640 10 monclient(hunting): dump: 2024-03-25T18:19:08.790 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:08.790 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:08.790 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:08.790 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:08.790 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:08.790 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:08.790 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:08.790 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:08.791 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:08.791 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:08.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.788+0000 7ff4477fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:08.797 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.796+0000 7ff4477fe640 10 client.?.objecter ms_handle_connect 0x7ff46013f710 2024-03-25T18:19:08.797 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.796+0000 7ff4477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:08.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.799+0000 7ff4477fe640 10 client.?.objecter ms_handle_connect 0x7ff4601430c0 2024-03-25T18:19:08.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.799+0000 7ff4477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:08.814 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.813+0000 7ff4477fe640 1 -- 172.21.15.122:0/4105144349 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623121051 0 0) 0x7ff4600ab9e0 con 0x7ff46013f710 2024-03-25T18:19:08.815 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.813+0000 7ff4477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.815 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.813+0000 7ff4477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.815 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.813+0000 7ff4477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.815 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.813+0000 7ff4477fe640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:19:08.815 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.813+0000 7ff4477fe640 1 -- 172.21.15.122:0/4105144349 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff44c004eb0 con 0x7ff46013f710 2024-03-25T18:19:08.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4477fe640 1 -- 172.21.15.122:0/4105144349 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 700875911 0 0) 0x7ff4601d7830 con 0x7ff4600a98f0 2024-03-25T18:19:08.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4477fe640 1 -- 172.21.15.122:0/4105144349 >> v1:172.21.15.122:6790/0 conn(0x7ff4601430c0 legacy=0x7ff4601e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4477fe640 1 -- 172.21.15.122:0/4105144349 >> v1:172.21.15.204:6789/0 conn(0x7ff46013f710 legacy=0x7ff4601d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4477fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:08.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4477fe640 1 monclient: found mon.a 2024-03-25T18:19:08.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:08.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4477fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:08.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4477fe640 1 -- 172.21.15.122:0/4105144349 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4601d83d0 con 0x7ff4600a98f0 2024-03-25T18:19:08.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4477fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:08.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4477fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:08.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:08.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4654e6640 5 monclient: authenticate success, global_id 4350 2024-03-25T18:19:08.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4654e6640 10 monclient: _renew_subs 2024-03-25T18:19:08.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4654e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:08.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4654e6640 1 -- 172.21.15.122:0/4105144349 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4601379c0 con 0x7ff4600a98f0 2024-03-25T18:19:08.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4654e6640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:08.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4654e6640 10 monclient: _renew_subs 2024-03-25T18:19:08.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4654e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:08.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4654e6640 1 -- 172.21.15.122:0/4105144349 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4601351f0 con 0x7ff4600a98f0 2024-03-25T18:19:08.821 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.817+0000 7ff4654e6640 1 librados: init done 2024-03-25T18:19:08.821 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.818+0000 7ff4654e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:08.821 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.818+0000 7ff4654e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:08.821 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.818+0000 7ff4654e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:08.822 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:08.818+0000 7ff4654e6640 1 -- 172.21.15.122:0/4105144349 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4600627b0 con 0x7ff4600a98f0 2024-03-25T18:19:09.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.118+0000 7ff4477fe640 1 -- 172.21.15.122:0/4105144349 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff450001b70 con 0x7ff4600a98f0 2024-03-25T18:19:09.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.118+0000 7ff4477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:09.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.118+0000 7ff4477fe640 1 -- 172.21.15.122:0/4105144349 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff450003a60 con 0x7ff4600a98f0 2024-03-25T18:19:09.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.118+0000 7ff4477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:09.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.118+0000 7ff4477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:09.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.118+0000 7ff4477fe640 10 monclient: dump: 2024-03-25T18:19:09.120 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:09.120 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:09.120 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:09.120 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:09.120 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:09.121 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:09.121 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:09.121 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:09.121 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:09.121 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:09.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.118+0000 7ff45d7fa640 4 set_mon_vals no callback set 2024-03-25T18:19:09.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.119+0000 7ff4477fe640 1 -- 172.21.15.122:0/4105144349 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7ff450003c80 con 0x7ff4600a98f0 2024-03-25T18:19:09.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.119+0000 7ff45e7fc640 10 monclient: get_auth_request con 0x7ff44c06dd40 auth_method 0 2024-03-25T18:19:09.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.120+0000 7ff4477fe640 10 client.4350.objecter ms_handle_connect 0x7ff44c06dd40 2024-03-25T18:19:09.125 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.124+0000 7ff4477fe640 1 -- 172.21.15.122:0/4105144349 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7ff450054930 con 0x7ff4600a98f0 2024-03-25T18:19:09.125 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.124+0000 7ff4477fe640 10 client.4350.objecter ms_dispatch 0x7ff460000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:09.125 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.124+0000 7ff4477fe640 3 client.4350.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:09.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.124+0000 7ff4477fe640 3 client.4350.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:09.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.124+0000 7ff4477fe640 20 client.4350.objecter dump_active .. 0 homeless 2024-03-25T18:19:09.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.124+0000 7ff4477fe640 1 -- 172.21.15.122:0/4105144349 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff450050440 con 0x7ff4600a98f0 2024-03-25T18:19:09.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.124+0000 7ff4477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:09.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.124+0000 7ff4477fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:09.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.352+0000 7ff4654e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:09.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.352+0000 7ff4654e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:09.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.353+0000 7ff4654e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:09.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.353+0000 7ff4654e6640 1 -- 172.21.15.122:0/4105144349 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff46014f240 con 0x7ff4600a98f0 2024-03-25T18:19:09.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.413+0000 7ff4477fe640 1 -- 172.21.15.122:0/4105144349 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7ff450052370 con 0x7ff4600a98f0 2024-03-25T18:19:09.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.413+0000 7ff4477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:09.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.413+0000 7ff4477fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:09.415 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:19:09.415 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:09.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.416+0000 7ff4654e6640 10 librados: watch_flush enter 2024-03-25T18:19:09.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.416+0000 7ff4654e6640 10 librados: watch_flush exit 2024-03-25T18:19:09.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.416+0000 7ff4654e6640 20 client.4350.objecter shutdown clearing up homeless session... 2024-03-25T18:19:09.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.416+0000 7ff4654e6640 10 client.4350.objecter successfully canceled tick 2024-03-25T18:19:09.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.416+0000 7ff4654e6640 1 -- 172.21.15.122:0/4105144349 >> v1:172.21.15.204:6800/676228005 conn(0x7ff44c06dd40 legacy=0x7ff44c070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:09.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.416+0000 7ff4654e6640 10 monclient: shutdown 2024-03-25T18:19:09.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.416+0000 7ff4654e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:09.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.416+0000 7ff4654e6640 1 -- 172.21.15.122:0/4105144349 >> v1:172.21.15.122:6789/0 conn(0x7ff4600a98f0 legacy=0x7ff460138760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:09.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.416+0000 7ff4654e6640 1 -- 172.21.15.122:0/4105144349 shutdown_connections 2024-03-25T18:19:09.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.416+0000 7ff4654e6640 1 -- 172.21.15.122:0/4105144349 >> 172.21.15.122:0/4105144349 conn(0x7ff4600b7ea0 msgr2=0x7ff460074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:09.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.416+0000 7ff4654e6640 1 -- 172.21.15.122:0/4105144349 shutdown_connections 2024-03-25T18:19:09.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.416+0000 7ff4654e6640 1 -- 172.21.15.122:0/4105144349 wait complete. 2024-03-25T18:19:09.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:09.416+0000 7ff4654e6640 1 librados: shutdown 2024-03-25T18:19:09.429 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:16.235 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:16.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.349+0000 7f83cbe25640 10 monclient: get_monmap_and_config 2024-03-25T18:19:16.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.349+0000 7f83cbe25640 10 monclient: build_initial_monmap 2024-03-25T18:19:16.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.349+0000 7f83cbe25640 10 monclient: monmap: 2024-03-25T18:19:16.350 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:19:16.350 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:16.350 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:19:16.350290+0000 2024-03-25T18:19:16.350 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:19:16.350290+0000 2024-03-25T18:19:16.350 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:16.351 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:16.351 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:19:16.351 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:19:16.351 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:19:16.351 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:16.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.350+0000 7f83cbe25640 1 Processor -- start 2024-03-25T18:19:16.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.350+0000 7f83cbe25640 1 -- start start 2024-03-25T18:19:16.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.350+0000 7f83cbe25640 10 monclient: init 2024-03-25T18:19:16.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.351+0000 7f83cbe25640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:16.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.351+0000 7f83cbe25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:16.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.351+0000 7f83cbe25640 10 monclient(hunting): picked mon.noname-b con 0x7f83c40a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:19:16.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.351+0000 7f83cbe25640 10 monclient(hunting): picked mon.noname-c con 0x7f83c413f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:19:16.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.351+0000 7f83cbe25640 10 monclient(hunting): picked mon.noname-a con 0x7f83c41430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:19:16.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.351+0000 7f83cbe25640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c4142520 con 0x7f83c41430c0 2024-03-25T18:19:16.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.351+0000 7f83cbe25640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c40b28a0 con 0x7f83c413f710 2024-03-25T18:19:16.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.351+0000 7f83cbe25640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c40b2a30 con 0x7f83c40a98f0 2024-03-25T18:19:16.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.351+0000 7f83cbe25640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:16.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.351+0000 7f83cbe25640 10 monclient(hunting): authenticate will time out at 7479.617432s 2024-03-25T18:19:16.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.363+0000 7f83c9bc1640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f83c40a98f0 0x7f83c40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:57548/0 (socket says 172.21.15.122:57548) 2024-03-25T18:19:16.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.363+0000 7f83c9bc1640 1 -- 172.21.15.122:0/460567763 learned_addr learned my addr 172.21.15.122:0/460567763 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:19:16.382 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.382+0000 7f83c8bbf640 1 -- 172.21.15.122:0/460567763 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462109830 0 0) 0x7f83c40b28a0 con 0x7f83c413f710 2024-03-25T18:19:16.382 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.382+0000 7f83c8bbf640 10 monclient(hunting): _negotiate 2024-03-25T18:19:16.382 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.382+0000 7f83c8bbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:16.382 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.382+0000 7f83c8bbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:16.382 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.382+0000 7f83c8bbf640 10 monclient(hunting): my global_id is 4351 2024-03-25T18:19:16.382 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.382+0000 7f83c8bbf640 1 -- 172.21.15.122:0/460567763 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b0003520 con 0x7f83c413f710 2024-03-25T18:19:16.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.383+0000 7f83c8bbf640 1 -- 172.21.15.122:0/460567763 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215495669 0 0) 0x7f83c4142520 con 0x7f83c41430c0 2024-03-25T18:19:16.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.383+0000 7f83c8bbf640 10 monclient(hunting): _negotiate 2024-03-25T18:19:16.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.383+0000 7f83c8bbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:16.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.383+0000 7f83c8bbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:16.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.383+0000 7f83c8bbf640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:19:16.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.383+0000 7f83c8bbf640 1 -- 172.21.15.122:0/460567763 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83c40b28a0 con 0x7f83c41430c0 2024-03-25T18:19:16.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.385+0000 7f83c8bbf640 1 -- 172.21.15.122:0/460567763 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3929381121 0 0) 0x7f83c40b2a30 con 0x7f83c40a98f0 2024-03-25T18:19:16.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.385+0000 7f83c8bbf640 10 monclient(hunting): _negotiate 2024-03-25T18:19:16.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.385+0000 7f83c8bbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:16.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.385+0000 7f83c8bbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:16.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.385+0000 7f83c8bbf640 10 monclient(hunting): my global_id is 4331 2024-03-25T18:19:16.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.385+0000 7f83c8bbf640 1 -- 172.21.15.122:0/460567763 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83c4142520 con 0x7f83c40a98f0 2024-03-25T18:19:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.396+0000 7f83c8bbf640 1 -- 172.21.15.122:0/460567763 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1580220844 0 0) 0x7f83c4142520 con 0x7f83c40a98f0 2024-03-25T18:19:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.396+0000 7f83c8bbf640 1 -- 172.21.15.122:0/460567763 >> v1:172.21.15.204:6789/0 conn(0x7f83c413f710 legacy=0x7f83c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.396+0000 7f83c8bbf640 1 -- 172.21.15.122:0/460567763 >> v1:172.21.15.122:6789/0 conn(0x7f83c41430c0 legacy=0x7f83c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.396+0000 7f83c8bbf640 10 monclient: _finish_hunting 0 2024-03-25T18:19:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.396+0000 7f83c8bbf640 1 monclient: found mon.noname-b 2024-03-25T18:19:16.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.396+0000 7f83c8bbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:16.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.396+0000 7f83c8bbf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:19:16.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.396+0000 7f83c8bbf640 1 -- 172.21.15.122:0/460567763 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c40b44c0 con 0x7f83c40a98f0 2024-03-25T18:19:16.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.396+0000 7f83c8bbf640 10 monclient: _finish_auth 0 2024-03-25T18:19:16.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.396+0000 7f83c8bbf640 10 monclient: _check_auth_tickets 2024-03-25T18:19:16.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.396+0000 7f83c8bbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:16.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.396+0000 7f83c8bbf640 1 -- 172.21.15.122:0/460567763 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f83c00015b0 con 0x7f83c40a98f0 2024-03-25T18:19:16.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.396+0000 7f83c8bbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:16.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.396+0000 7f83c8bbf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:16.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.396+0000 7f83c8bbf640 10 monclient: dump: 2024-03-25T18:19:16.397 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:16.397 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:16.397 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:16.397 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:16.397 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:16.397 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:16.398 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:16.398 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:16.398 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:16.398 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:16.398 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.396+0000 7f83cbe25640 5 monclient: authenticate success, global_id 4331 2024-03-25T18:19:16.398 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.396+0000 7f83cbe25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:16.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.408+0000 7f83c8bbf640 1 -- 172.21.15.122:0/460567763 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83c0001710 con 0x7f83c40a98f0 2024-03-25T18:19:16.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.408+0000 7f83c8bbf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:16.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.408+0000 7f83c8bbf640 1 -- 172.21.15.122:0/460567763 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f83c0001b20 con 0x7f83c40a98f0 2024-03-25T18:19:16.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.408+0000 7f83c8bbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:16.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.408+0000 7f83c8bbf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:16.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.408+0000 7f83c8bbf640 10 monclient: dump: 2024-03-25T18:19:16.409 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:16.409 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:16.409 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:16.409 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:16.409 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:16.409 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:16.409 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:16.410 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:16.410 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:16.410 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:16.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.408+0000 7f83cbe25640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:16.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.408+0000 7f83cbe25640 4 set_mon_vals no callback set 2024-03-25T18:19:16.412 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.409+0000 7f83cbe25640 10 monclient: shutdown 2024-03-25T18:19:16.412 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.409+0000 7f83cbe25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:16.412 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.409+0000 7f83cbe25640 1 -- 172.21.15.122:0/460567763 >> v1:172.21.15.122:6790/0 conn(0x7f83c40a98f0 legacy=0x7f83c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.412 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.409+0000 7f83cbe25640 1 -- 172.21.15.122:0/460567763 shutdown_connections 2024-03-25T18:19:16.412 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.409+0000 7f83cbe25640 1 -- 172.21.15.122:0/460567763 >> 172.21.15.122:0/460567763 conn(0x7f83c40b7ea0 msgr2=0x7f83c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:16.412 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.410+0000 7f83cbe25640 1 -- 172.21.15.122:0/460567763 shutdown_connections 2024-03-25T18:19:16.412 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.410+0000 7f83cbe25640 1 -- 172.21.15.122:0/460567763 wait complete. 2024-03-25T18:19:16.412 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.410+0000 7f83cbe25640 10 monclient: build_initial_monmap 2024-03-25T18:19:16.412 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.410+0000 7f83cbe25640 10 monclient: monmap: 2024-03-25T18:19:16.412 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:19:16.412 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:16.412 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:19:16.412 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:19:16.412 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:16.413 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:19:16.413 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:19:16.413 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:19:16.413 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:19:16.413 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:16.413 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.411+0000 7f83cbe25640 1 librados: starting msgr at 2024-03-25T18:19:16.413 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.411+0000 7f83cbe25640 1 librados: starting objecter 2024-03-25T18:19:16.413 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.411+0000 7f83cbe25640 1 Processor -- start 2024-03-25T18:19:16.413 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.411+0000 7f83cbe25640 1 -- start start 2024-03-25T18:19:16.413 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.411+0000 7f83cbe25640 1 librados: setting wanted keys 2024-03-25T18:19:16.413 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.411+0000 7f83cbe25640 1 librados: calling monclient init 2024-03-25T18:19:16.413 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.411+0000 7f83cbe25640 10 monclient: init 2024-03-25T18:19:16.413 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.411+0000 7f83cbe25640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:16.413 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.412+0000 7f83cbe25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:16.413 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.412+0000 7f83cbe25640 10 monclient(hunting): picked mon.noname-c con 0x7f83c40a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:19:16.413 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.412+0000 7f83cbe25640 10 monclient(hunting): picked mon.noname-a con 0x7f83c413f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:19:16.413 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.412+0000 7f83cbe25640 10 monclient(hunting): picked mon.noname-b con 0x7f83c41430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:19:16.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.412+0000 7f83cbe25640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c414e910 con 0x7f83c413f710 2024-03-25T18:19:16.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.412+0000 7f83cbe25640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c40abb90 con 0x7f83c41430c0 2024-03-25T18:19:16.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.412+0000 7f83cbe25640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83c414e5c0 con 0x7f83c40a98f0 2024-03-25T18:19:16.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.412+0000 7f83cbe25640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:16.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.412+0000 7f83cbe25640 10 monclient(hunting): authenticate will time out at 7479.677065s 2024-03-25T18:19:16.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.420+0000 7f83c9bc1640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f83c40a98f0 0x7f83c41383c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:57600/0 (socket says 172.21.15.122:57600) 2024-03-25T18:19:16.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.420+0000 7f83c9bc1640 1 -- 172.21.15.122:0/2525943480 learned_addr learned my addr 172.21.15.122:0/2525943480 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:19:16.421 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.421+0000 7f83ba7fc640 10 client.?.objecter ms_handle_connect 0x7f83c413f710 2024-03-25T18:19:16.421 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.421+0000 7f83ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:16.425 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.425+0000 7f83ba7fc640 10 client.?.objecter ms_handle_connect 0x7f83c40a98f0 2024-03-25T18:19:16.425 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.425+0000 7f83ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:16.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.435+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157788405 0 0) 0x7f83c414e910 con 0x7f83c413f710 2024-03-25T18:19:16.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.435+0000 7f83ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:16.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.435+0000 7f83ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:16.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.435+0000 7f83ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:16.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.435+0000 7f83ba7fc640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:19:16.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.435+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83a8004eb0 con 0x7f83c413f710 2024-03-25T18:19:16.437 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.437+0000 7f83ba7fc640 10 client.?.objecter ms_handle_connect 0x7f83c41430c0 2024-03-25T18:19:16.437 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.437+0000 7f83ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:16.446 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.445+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2783385531 0 0) 0x7f83c414e5c0 con 0x7f83c40a98f0 2024-03-25T18:19:16.446 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.446+0000 7f83ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:16.446 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.446+0000 7f83ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:16.446 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.446+0000 7f83ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:16.446 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.446+0000 7f83ba7fc640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:19:16.446 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.446+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83c414e910 con 0x7f83c40a98f0 2024-03-25T18:19:16.457 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.456+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1693741359 0 0) 0x7f83a8004eb0 con 0x7f83c413f710 2024-03-25T18:19:16.457 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.456+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83c414e5c0 con 0x7f83c413f710 2024-03-25T18:19:16.457 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.457+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f83b4001950 con 0x7f83c413f710 2024-03-25T18:19:16.457 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.457+0000 7f83ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:16.457 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.457+0000 7f83ba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:16.457 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.457+0000 7f83ba7fc640 10 monclient(hunting): dump: 2024-03-25T18:19:16.457 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:16.457 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:16.457 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:16.457 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:16.458 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:16.458 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:16.458 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:16.458 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:16.458 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:16.458 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:16.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.457+0000 7f83ba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:16.462 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.462+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3245248064 0 0) 0x7f83c40abb90 con 0x7f83c41430c0 2024-03-25T18:19:16.462 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.462+0000 7f83ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:16.462 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.462+0000 7f83ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:16.462 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.462+0000 7f83ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:16.462 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.462+0000 7f83ba7fc640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:19:16.462 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.462+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83a8004eb0 con 0x7f83c41430c0 2024-03-25T18:19:16.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.467+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2485317528 0 0) 0x7f83c414e5c0 con 0x7f83c413f710 2024-03-25T18:19:16.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.467+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 >> v1:172.21.15.122:6790/0 conn(0x7f83c40a98f0 legacy=0x7f83c41383c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.467+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 >> v1:172.21.15.204:6789/0 conn(0x7f83c41430c0 legacy=0x7f83c41d7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.467+0000 7f83ba7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:16.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.467+0000 7f83ba7fc640 1 monclient: found mon.a 2024-03-25T18:19:16.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.467+0000 7f83ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:16.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.467+0000 7f83ba7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:16.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.467+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83c4138ad0 con 0x7f83c413f710 2024-03-25T18:19:16.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.467+0000 7f83ba7fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:16.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.467+0000 7f83ba7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:16.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.467+0000 7f83ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:16.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.467+0000 7f83cbe25640 5 monclient: authenticate success, global_id 4356 2024-03-25T18:19:16.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.468+0000 7f83cbe25640 10 monclient: _renew_subs 2024-03-25T18:19:16.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.468+0000 7f83cbe25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:16.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.468+0000 7f83cbe25640 1 -- 172.21.15.122:0/2525943480 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83c4138d70 con 0x7f83c413f710 2024-03-25T18:19:16.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.468+0000 7f83cbe25640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:16.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.468+0000 7f83cbe25640 10 monclient: _renew_subs 2024-03-25T18:19:16.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.468+0000 7f83cbe25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:16.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.468+0000 7f83cbe25640 1 -- 172.21.15.122:0/2525943480 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83c41e32a0 con 0x7f83c413f710 2024-03-25T18:19:16.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.468+0000 7f83cbe25640 1 librados: init done 2024-03-25T18:19:16.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.469+0000 7f83cbe25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:16.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.469+0000 7f83cbe25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:16.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.469+0000 7f83cbe25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:16.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.469+0000 7f83cbe25640 1 -- 172.21.15.122:0/2525943480 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83c40627b0 con 0x7f83c413f710 2024-03-25T18:19:16.782 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.782+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83b4003610 con 0x7f83c413f710 2024-03-25T18:19:16.782 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.782+0000 7f83ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:16.782 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.782+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f83b4003920 con 0x7f83c413f710 2024-03-25T18:19:16.782 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.782+0000 7f83ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:16.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.782+0000 7f83ba7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:16.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.782+0000 7f83ba7fc640 10 monclient: dump: 2024-03-25T18:19:16.783 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:16.783 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:16.783 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:16.783 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:16.783 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:16.783 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:16.783 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:16.783 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:16.783 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:16.783 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:16.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.782+0000 7f83bbfff640 4 set_mon_vals no callback set 2024-03-25T18:19:16.784 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.783+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f83b40198f0 con 0x7f83c413f710 2024-03-25T18:19:16.784 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.784+0000 7f83c9bc1640 10 monclient: get_auth_request con 0x7f83a806ddb0 auth_method 0 2024-03-25T18:19:16.784 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.784+0000 7f83ba7fc640 10 client.4356.objecter ms_handle_connect 0x7f83a806ddb0 2024-03-25T18:19:16.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.789+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f83b40535b0 con 0x7f83c413f710 2024-03-25T18:19:16.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.789+0000 7f83ba7fc640 10 client.4356.objecter ms_dispatch 0x7f83c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:16.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.789+0000 7f83ba7fc640 3 client.4356.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:16.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.789+0000 7f83ba7fc640 3 client.4356.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:16.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.789+0000 7f83ba7fc640 20 client.4356.objecter dump_active .. 0 homeless 2024-03-25T18:19:16.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.789+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f83b404eef0 con 0x7f83c413f710 2024-03-25T18:19:16.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.789+0000 7f83ba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:16.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:16.789+0000 7f83ba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:17.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.029+0000 7f83cbe25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:17.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.029+0000 7f83cbe25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:17.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.029+0000 7f83cbe25640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:17.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.029+0000 7f83cbe25640 1 -- 172.21.15.122:0/2525943480 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83c4141c20 con 0x7f83c413f710 2024-03-25T18:19:17.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.054+0000 7f83ba7fc640 1 -- 172.21.15.122:0/2525943480 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f83b4050ff0 con 0x7f83c413f710 2024-03-25T18:19:17.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.055+0000 7f83ba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:17.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.055+0000 7f83ba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:17.055 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:19:17.055 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:17.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.056+0000 7f83cbe25640 10 librados: watch_flush enter 2024-03-25T18:19:17.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.057+0000 7f83cbe25640 10 librados: watch_flush exit 2024-03-25T18:19:17.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.057+0000 7f83cbe25640 20 client.4356.objecter shutdown clearing up homeless session... 2024-03-25T18:19:17.058 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.057+0000 7f83cbe25640 10 client.4356.objecter successfully canceled tick 2024-03-25T18:19:17.058 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.057+0000 7f83cbe25640 1 -- 172.21.15.122:0/2525943480 >> v1:172.21.15.204:6800/676228005 conn(0x7f83a806ddb0 legacy=0x7f83a8070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.058 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.057+0000 7f83cbe25640 10 monclient: shutdown 2024-03-25T18:19:17.058 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.057+0000 7f83cbe25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:17.058 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.057+0000 7f83cbe25640 1 -- 172.21.15.122:0/2525943480 >> v1:172.21.15.122:6789/0 conn(0x7f83c413f710 legacy=0x7f83c41d7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.058 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.057+0000 7f83cbe25640 1 -- 172.21.15.122:0/2525943480 shutdown_connections 2024-03-25T18:19:17.058 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.057+0000 7f83cbe25640 1 -- 172.21.15.122:0/2525943480 >> 172.21.15.122:0/2525943480 conn(0x7f83c40b7ea0 msgr2=0x7f83c4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:17.058 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.057+0000 7f83cbe25640 1 -- 172.21.15.122:0/2525943480 shutdown_connections 2024-03-25T18:19:17.058 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.057+0000 7f83cbe25640 1 -- 172.21.15.122:0/2525943480 wait complete. 2024-03-25T18:19:17.058 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:17.057+0000 7f83cbe25640 1 librados: shutdown 2024-03-25T18:19:17.067 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:23.874 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:23.993 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:23.992+0000 7fdb538a8640 10 monclient: get_monmap_and_config 2024-03-25T18:19:23.993 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:23.992+0000 7fdb538a8640 10 monclient: build_initial_monmap 2024-03-25T18:19:23.993 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:23.992+0000 7fdb538a8640 10 monclient: monmap: 2024-03-25T18:19:23.993 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:19:23.993192+0000 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:19:23.993192+0000 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:19:23.994 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:23.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:23.992+0000 7fdb538a8640 1 Processor -- start 2024-03-25T18:19:23.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:23.993+0000 7fdb538a8640 1 -- start start 2024-03-25T18:19:23.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:23.993+0000 7fdb538a8640 10 monclient: init 2024-03-25T18:19:23.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:23.993+0000 7fdb538a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:23.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:23.993+0000 7fdb538a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:23.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:23.993+0000 7fdb538a8640 10 monclient(hunting): picked mon.noname-c con 0x7fdb4c0a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:19:23.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:23.993+0000 7fdb538a8640 10 monclient(hunting): picked mon.noname-b con 0x7fdb4c13f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:19:23.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:23.993+0000 7fdb538a8640 10 monclient(hunting): picked mon.noname-a con 0x7fdb4c1430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:19:23.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:23.993+0000 7fdb538a8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4c142520 con 0x7fdb4c1430c0 2024-03-25T18:19:23.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:23.993+0000 7fdb538a8640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4c0b28a0 con 0x7fdb4c0a98f0 2024-03-25T18:19:23.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:23.993+0000 7fdb538a8640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4c0b2a30 con 0x7fdb4c13f710 2024-03-25T18:19:23.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:23.993+0000 7fdb538a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:23.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:23.993+0000 7fdb538a8640 10 monclient(hunting): authenticate will time out at 7487.259891s 2024-03-25T18:19:24.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.003+0000 7fdb51644640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fdb4c0a98f0 0x7fdb4c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:49670/0 (socket says 172.21.15.122:49670) 2024-03-25T18:19:24.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.003+0000 7fdb51644640 1 -- 172.21.15.122:0/1122167735 learned_addr learned my addr 172.21.15.122:0/1122167735 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:19:24.016 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.015+0000 7fdb43fff640 1 -- 172.21.15.122:0/1122167735 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395689064 0 0) 0x7fdb4c0b2a30 con 0x7fdb4c13f710 2024-03-25T18:19:24.016 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.015+0000 7fdb43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:24.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.015+0000 7fdb43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:24.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.015+0000 7fdb43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:24.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.015+0000 7fdb43fff640 10 monclient(hunting): my global_id is 4337 2024-03-25T18:19:24.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.015+0000 7fdb43fff640 1 -- 172.21.15.122:0/1122167735 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb38003520 con 0x7fdb4c13f710 2024-03-25T18:19:24.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.023+0000 7fdb43fff640 1 -- 172.21.15.122:0/1122167735 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 982615990 0 0) 0x7fdb4c0b28a0 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.023+0000 7fdb43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:24.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.023+0000 7fdb43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:24.025 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.023+0000 7fdb43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:24.025 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.023+0000 7fdb43fff640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:19:24.025 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.023+0000 7fdb43fff640 1 -- 172.21.15.122:0/1122167735 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb4c0b2a30 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.031+0000 7fdb43fff640 1 -- 172.21.15.122:0/1122167735 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189956400 0 0) 0x7fdb4c142520 con 0x7fdb4c1430c0 2024-03-25T18:19:24.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.031+0000 7fdb43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:24.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.031+0000 7fdb43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:24.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.031+0000 7fdb43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:24.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.031+0000 7fdb43fff640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:19:24.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.031+0000 7fdb43fff640 1 -- 172.21.15.122:0/1122167735 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb4c0b28a0 con 0x7fdb4c1430c0 2024-03-25T18:19:24.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.035+0000 7fdb43fff640 1 -- 172.21.15.122:0/1122167735 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2935576360 0 0) 0x7fdb4c0b2a30 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.035+0000 7fdb43fff640 1 -- 172.21.15.122:0/1122167735 >> v1:172.21.15.122:6790/0 conn(0x7fdb4c13f710 legacy=0x7fdb4c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.035+0000 7fdb43fff640 1 -- 172.21.15.122:0/1122167735 >> v1:172.21.15.122:6789/0 conn(0x7fdb4c1430c0 legacy=0x7fdb4c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.035+0000 7fdb43fff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:24.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.035+0000 7fdb43fff640 1 monclient: found mon.noname-c 2024-03-25T18:19:24.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.035+0000 7fdb43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:24.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.035+0000 7fdb43fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:19:24.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.035+0000 7fdb43fff640 1 -- 172.21.15.122:0/1122167735 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4c0b44c0 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.035+0000 7fdb43fff640 10 monclient: _finish_auth 0 2024-03-25T18:19:24.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.035+0000 7fdb43fff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:24.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.035+0000 7fdb43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:24.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.035+0000 7fdb43fff640 1 -- 172.21.15.122:0/1122167735 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fdb48001670 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.039 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.036+0000 7fdb538a8640 5 monclient: authenticate success, global_id 4357 2024-03-25T18:19:24.039 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.036+0000 7fdb538a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:24.039 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.036+0000 7fdb43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:24.039 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.036+0000 7fdb43fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:24.039 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.036+0000 7fdb43fff640 10 monclient: dump: 2024-03-25T18:19:24.039 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:24.039 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:24.039 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:24.039 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:24.039 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:24.040 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:24.040 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:24.040 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:24.040 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:24.040 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:24.040 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.036+0000 7fdb538a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:24.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.047+0000 7fdb43fff640 1 -- 172.21.15.122:0/1122167735 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb480017d0 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.047+0000 7fdb43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:24.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.047+0000 7fdb43fff640 1 -- 172.21.15.122:0/1122167735 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fdb48001be0 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.047+0000 7fdb538a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:24.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.047+0000 7fdb538a8640 4 set_mon_vals no callback set 2024-03-25T18:19:24.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.047+0000 7fdb43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:24.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.047+0000 7fdb43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:24.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.047+0000 7fdb43fff640 10 monclient: dump: 2024-03-25T18:19:24.049 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:24.049 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:24.049 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:24.049 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:24.049 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:24.049 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:24.049 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:24.049 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:24.049 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:24.049 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:24.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.047+0000 7fdb538a8640 10 monclient: shutdown 2024-03-25T18:19:24.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.047+0000 7fdb538a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:24.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.047+0000 7fdb538a8640 1 -- 172.21.15.122:0/1122167735 >> v1:172.21.15.204:6789/0 conn(0x7fdb4c0a98f0 legacy=0x7fdb4c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.047+0000 7fdb538a8640 1 -- 172.21.15.122:0/1122167735 shutdown_connections 2024-03-25T18:19:24.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.047+0000 7fdb538a8640 1 -- 172.21.15.122:0/1122167735 >> 172.21.15.122:0/1122167735 conn(0x7fdb4c0b7ea0 msgr2=0x7fdb4c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:24.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.048+0000 7fdb538a8640 1 -- 172.21.15.122:0/1122167735 shutdown_connections 2024-03-25T18:19:24.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.048+0000 7fdb538a8640 1 -- 172.21.15.122:0/1122167735 wait complete. 2024-03-25T18:19:24.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.048+0000 7fdb538a8640 10 monclient: build_initial_monmap 2024-03-25T18:19:24.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.048+0000 7fdb538a8640 10 monclient: monmap: 2024-03-25T18:19:24.053 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:19:24.053 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:24.053 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:19:24.053 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:19:24.053 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:24.053 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:19:24.053 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:19:24.053 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:19:24.053 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:19:24.054 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:24.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.049+0000 7fdb538a8640 1 librados: starting msgr at 2024-03-25T18:19:24.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.049+0000 7fdb538a8640 1 librados: starting objecter 2024-03-25T18:19:24.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.049+0000 7fdb538a8640 1 Processor -- start 2024-03-25T18:19:24.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.049+0000 7fdb538a8640 1 -- start start 2024-03-25T18:19:24.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.049+0000 7fdb538a8640 1 librados: setting wanted keys 2024-03-25T18:19:24.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.049+0000 7fdb538a8640 1 librados: calling monclient init 2024-03-25T18:19:24.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.049+0000 7fdb538a8640 10 monclient: init 2024-03-25T18:19:24.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.050+0000 7fdb538a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:24.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.050+0000 7fdb538a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:24.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.050+0000 7fdb538a8640 10 monclient(hunting): picked mon.noname-c con 0x7fdb4c0a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:19:24.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.050+0000 7fdb538a8640 10 monclient(hunting): picked mon.noname-b con 0x7fdb4c13f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:19:24.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.050+0000 7fdb538a8640 10 monclient(hunting): picked mon.noname-a con 0x7fdb4c1430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:19:24.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.050+0000 7fdb538a8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4c1d51f0 con 0x7fdb4c1430c0 2024-03-25T18:19:24.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.050+0000 7fdb538a8640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4c1d83d0 con 0x7fdb4c13f710 2024-03-25T18:19:24.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.050+0000 7fdb538a8640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb4c1d77a0 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.050+0000 7fdb538a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:24.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.050+0000 7fdb538a8640 10 monclient(hunting): authenticate will time out at 7487.316192s 2024-03-25T18:19:24.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.059+0000 7fdb51644640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7fdb4c0a98f0 0x7fdb4c1d7cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:40388/0 (socket says 172.21.15.122:40388) 2024-03-25T18:19:24.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.059+0000 7fdb51644640 1 -- 172.21.15.122:0/1395621979 learned_addr learned my addr 172.21.15.122:0/1395621979 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:19:24.065 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.064+0000 7fdb41ffb640 10 client.?.objecter ms_handle_connect 0x7fdb4c0a98f0 2024-03-25T18:19:24.065 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.064+0000 7fdb41ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:24.068 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.067+0000 7fdb41ffb640 10 client.?.objecter ms_handle_connect 0x7fdb4c13f710 2024-03-25T18:19:24.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.067+0000 7fdb41ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:24.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.073+0000 7fdb41ffb640 10 client.?.objecter ms_handle_connect 0x7fdb4c1430c0 2024-03-25T18:19:24.075 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.073+0000 7fdb41ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:24.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.078+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257534154 0 0) 0x7fdb4c1d77a0 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.078+0000 7fdb41ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:24.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.078+0000 7fdb41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:24.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.078+0000 7fdb41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:24.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.078+0000 7fdb41ffb640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:19:24.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.078+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb30004eb0 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.081 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.079+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802258421 0 0) 0x7fdb4c1d83d0 con 0x7fdb4c13f710 2024-03-25T18:19:24.081 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.079+0000 7fdb41ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:24.081 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.079+0000 7fdb41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:24.081 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.079+0000 7fdb41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:24.081 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.079+0000 7fdb41ffb640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:19:24.081 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.079+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb4c1d77a0 con 0x7fdb4c13f710 2024-03-25T18:19:24.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.084+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1848768247 0 0) 0x7fdb4c1d51f0 con 0x7fdb4c1430c0 2024-03-25T18:19:24.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.084+0000 7fdb41ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:24.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.084+0000 7fdb41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:24.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.084+0000 7fdb41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:24.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.084+0000 7fdb41ffb640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:19:24.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.084+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb4c1d83d0 con 0x7fdb4c1430c0 2024-03-25T18:19:24.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.092+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1388089330 0 0) 0x7fdb4c1d77a0 con 0x7fdb4c13f710 2024-03-25T18:19:24.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.092+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb4c1d51f0 con 0x7fdb4c13f710 2024-03-25T18:19:24.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.092+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fdb3c001950 con 0x7fdb4c13f710 2024-03-25T18:19:24.094 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.092+0000 7fdb41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:24.094 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.092+0000 7fdb41ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:24.094 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.092+0000 7fdb41ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:24.094 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:24.094 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:24.094 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:24.094 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:24.094 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:24.094 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:24.094 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:24.095 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:24.095 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:24.095 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:24.095 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.092+0000 7fdb41ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:24.096 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.093+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3535315122 0 0) 0x7fdb30004eb0 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.096 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.093+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb4c1d77a0 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.096 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.093+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fdb48001be0 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.096 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.093+0000 7fdb41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:24.096 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.093+0000 7fdb41ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:24.096 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.093+0000 7fdb41ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:24.096 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:24.096 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:24.097 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:24.097 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:24.097 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:24.097 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:24.097 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:24.097 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:24.097 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:24.097 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:24.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.100+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2852553118 0 0) 0x7fdb4c1d77a0 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.100+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 >> v1:172.21.15.204:6789/0 conn(0x7fdb4c13f710 legacy=0x7fdb4c138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.100+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 >> v1:172.21.15.122:6789/0 conn(0x7fdb4c1430c0 legacy=0x7fdb4c1e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.100+0000 7fdb41ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:24.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.100+0000 7fdb41ffb640 1 monclient: found mon.c 2024-03-25T18:19:24.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.100+0000 7fdb41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:24.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.100+0000 7fdb41ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:19:24.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.100+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb4c138d20 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.100+0000 7fdb41ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:24.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.100+0000 7fdb41ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:24.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.100+0000 7fdb41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:24.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.100+0000 7fdb538a8640 5 monclient: authenticate success, global_id 4340 2024-03-25T18:19:24.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.100+0000 7fdb538a8640 10 monclient: _renew_subs 2024-03-25T18:19:24.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.100+0000 7fdb538a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:19:24.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.100+0000 7fdb538a8640 1 -- 172.21.15.122:0/1395621979 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb4c138f70 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.101+0000 7fdb538a8640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:24.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.101+0000 7fdb538a8640 10 monclient: _renew_subs 2024-03-25T18:19:24.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.101+0000 7fdb538a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:19:24.104 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.101+0000 7fdb538a8640 1 -- 172.21.15.122:0/1395621979 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4c137ae0 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.104 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.101+0000 7fdb538a8640 1 librados: init done 2024-03-25T18:19:24.104 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.102+0000 7fdb538a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:24.105 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.102+0000 7fdb538a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:24.105 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.102+0000 7fdb538a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:19:24.105 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.102+0000 7fdb538a8640 1 -- 172.21.15.122:0/1395621979 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb4c1d7930 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.123 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.122+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb48004a90 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.123 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.122+0000 7fdb41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:24.123 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.122+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fdb48004de0 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.124 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.122+0000 7fdb41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:24.124 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.122+0000 7fdb41ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:24.124 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.122+0000 7fdb41ffb640 10 monclient: dump: 2024-03-25T18:19:24.124 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:24.124 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:24.124 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:24.124 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:24.124 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:24.124 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:24.125 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:24.125 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:24.125 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:24.125 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:24.125 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.122+0000 7fdb437fe640 4 set_mon_vals no callback set 2024-03-25T18:19:24.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.123+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fdb48005000 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.123+0000 7fdb50e43640 10 monclient: get_auth_request con 0x7fdb3006dfb0 auth_method 0 2024-03-25T18:19:24.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.124+0000 7fdb41ffb640 10 client.4340.objecter ms_handle_connect 0x7fdb3006dfb0 2024-03-25T18:19:24.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.127+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fdb480549b0 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.127+0000 7fdb41ffb640 10 client.4340.objecter ms_dispatch 0x7fdb4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:24.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.127+0000 7fdb41ffb640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:24.129 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.127+0000 7fdb41ffb640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:24.129 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.127+0000 7fdb41ffb640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-25T18:19:24.425 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.424+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb480504c0 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.425 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.424+0000 7fdb41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:24.425 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.424+0000 7fdb41ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:24.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.681+0000 7fdb538a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:24.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.681+0000 7fdb538a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:24.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.681+0000 7fdb538a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:19:24.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.681+0000 7fdb538a8640 1 -- 172.21.15.122:0/1395621979 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb4c0627b0 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.816+0000 7fdb41ffb640 1 -- 172.21.15.122:0/1395621979 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fdb480523f0 con 0x7fdb4c0a98f0 2024-03-25T18:19:24.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.816+0000 7fdb41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:24.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.816+0000 7fdb41ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:24.817 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:19:24.817 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:24.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.817+0000 7fdb538a8640 10 librados: watch_flush enter 2024-03-25T18:19:24.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.818+0000 7fdb538a8640 10 librados: watch_flush exit 2024-03-25T18:19:24.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.818+0000 7fdb538a8640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-25T18:19:24.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.818+0000 7fdb538a8640 10 client.4340.objecter successfully canceled tick 2024-03-25T18:19:24.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.818+0000 7fdb538a8640 1 -- 172.21.15.122:0/1395621979 >> v1:172.21.15.204:6800/676228005 conn(0x7fdb3006dfb0 legacy=0x7fdb30070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.818+0000 7fdb538a8640 10 monclient: shutdown 2024-03-25T18:19:24.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.818+0000 7fdb538a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:24.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.818+0000 7fdb538a8640 1 -- 172.21.15.122:0/1395621979 >> v1:172.21.15.122:6790/0 conn(0x7fdb4c0a98f0 legacy=0x7fdb4c1d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.818+0000 7fdb538a8640 1 -- 172.21.15.122:0/1395621979 shutdown_connections 2024-03-25T18:19:24.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.818+0000 7fdb538a8640 1 -- 172.21.15.122:0/1395621979 >> 172.21.15.122:0/1395621979 conn(0x7fdb4c0b7ea0 msgr2=0x7fdb4c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:24.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.818+0000 7fdb538a8640 1 -- 172.21.15.122:0/1395621979 shutdown_connections 2024-03-25T18:19:24.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.818+0000 7fdb538a8640 1 -- 172.21.15.122:0/1395621979 wait complete. 2024-03-25T18:19:24.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:24.818+0000 7fdb538a8640 1 librados: shutdown 2024-03-25T18:19:24.832 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:25.856 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:03:19.599 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:25.862 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:03:19.605 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:25.892 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:03:19.635 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:25.912 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:03:19.658 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:25.914 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:03:19.657 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:25.923 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:03:19.669 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:25.926 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:03:19.668 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:25.944 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:03:19.690 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:25.948 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:03:19.691 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:31.637 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:31.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.753+0000 7f8bb7f96640 10 monclient: get_monmap_and_config 2024-03-25T18:19:31.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.753+0000 7f8bb7f96640 10 monclient: build_initial_monmap 2024-03-25T18:19:31.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.753+0000 7f8bb7f96640 10 monclient: monmap: 2024-03-25T18:19:31.754 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:19:31.754 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:31.754 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:19:31.754251+0000 2024-03-25T18:19:31.754 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:19:31.754251+0000 2024-03-25T18:19:31.754 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:31.754 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:31.754 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:19:31.754 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:19:31.754 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:19:31.754 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:31.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.753+0000 7f8bb7f96640 1 Processor -- start 2024-03-25T18:19:31.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.754+0000 7f8bb7f96640 1 -- start start 2024-03-25T18:19:31.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.754+0000 7f8bb7f96640 10 monclient: init 2024-03-25T18:19:31.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.754+0000 7f8bb7f96640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:31.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.754+0000 7f8bb7f96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:31.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.754+0000 7f8bb7f96640 10 monclient(hunting): picked mon.noname-c con 0x7f8bb00a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:19:31.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.754+0000 7f8bb7f96640 10 monclient(hunting): picked mon.noname-a con 0x7f8bb013f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:19:31.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.754+0000 7f8bb7f96640 10 monclient(hunting): picked mon.noname-b con 0x7f8bb01430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:19:31.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.754+0000 7f8bb7f96640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb0142520 con 0x7f8bb013f710 2024-03-25T18:19:31.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.754+0000 7f8bb7f96640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb00b28a0 con 0x7f8bb00a98f0 2024-03-25T18:19:31.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.754+0000 7f8bb7f96640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb00b2a30 con 0x7f8bb01430c0 2024-03-25T18:19:31.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.754+0000 7f8bb7f96640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:31.756 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.754+0000 7f8bb7f96640 10 monclient(hunting): authenticate will time out at 7495.020604s 2024-03-25T18:19:31.759 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.758+0000 7f8bb5531640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f8bb013f710 0x7f8bb01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:34522/0 (socket says 172.21.15.122:34522) 2024-03-25T18:19:31.759 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.758+0000 7f8bb5531640 1 -- 172.21.15.122:0/2538567532 learned_addr learned my addr 172.21.15.122:0/2538567532 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:19:31.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.764+0000 7f8bb4d30640 1 -- 172.21.15.122:0/2538567532 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547870608 0 0) 0x7f8bb0142520 con 0x7f8bb013f710 2024-03-25T18:19:31.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.764+0000 7f8bb4d30640 10 monclient(hunting): _negotiate 2024-03-25T18:19:31.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.764+0000 7f8bb4d30640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:31.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.764+0000 7f8bb4d30640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:31.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.764+0000 7f8bb4d30640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:19:31.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.764+0000 7f8bb4d30640 1 -- 172.21.15.122:0/2538567532 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b9c003520 con 0x7f8bb013f710 2024-03-25T18:19:31.777 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.776+0000 7f8bb4d30640 1 -- 172.21.15.122:0/2538567532 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359406956 0 0) 0x7f8bb00b2a30 con 0x7f8bb01430c0 2024-03-25T18:19:31.777 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.776+0000 7f8bb4d30640 10 monclient(hunting): _negotiate 2024-03-25T18:19:31.777 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.776+0000 7f8bb4d30640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:31.777 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.776+0000 7f8bb4d30640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:31.777 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.776+0000 7f8bb4d30640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:19:31.777 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.776+0000 7f8bb4d30640 1 -- 172.21.15.122:0/2538567532 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb0142520 con 0x7f8bb01430c0 2024-03-25T18:19:31.781 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.780+0000 7f8bb4d30640 1 -- 172.21.15.122:0/2538567532 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 533162661 0 0) 0x7f8bb00b28a0 con 0x7f8bb00a98f0 2024-03-25T18:19:31.781 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.780+0000 7f8bb4d30640 10 monclient(hunting): _negotiate 2024-03-25T18:19:31.781 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.780+0000 7f8bb4d30640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:31.781 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.780+0000 7f8bb4d30640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:31.781 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.780+0000 7f8bb4d30640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:19:31.782 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.780+0000 7f8bb4d30640 1 -- 172.21.15.122:0/2538567532 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb00b2a30 con 0x7f8bb00a98f0 2024-03-25T18:19:31.782 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.781+0000 7f8bb4d30640 1 -- 172.21.15.122:0/2538567532 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 113279254 0 0) 0x7f8b9c003520 con 0x7f8bb013f710 2024-03-25T18:19:31.782 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.781+0000 7f8bb4d30640 1 -- 172.21.15.122:0/2538567532 >> v1:172.21.15.122:6790/0 conn(0x7f8bb01430c0 legacy=0x7f8bb014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.782 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.781+0000 7f8bb4d30640 1 -- 172.21.15.122:0/2538567532 >> v1:172.21.15.204:6789/0 conn(0x7f8bb00a98f0 legacy=0x7f8bb00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.782 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.781+0000 7f8bb4d30640 10 monclient: _finish_hunting 0 2024-03-25T18:19:31.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.781+0000 7f8bb4d30640 1 monclient: found mon.noname-a 2024-03-25T18:19:31.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.781+0000 7f8bb4d30640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:31.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.781+0000 7f8bb4d30640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:19:31.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.781+0000 7f8bb4d30640 1 -- 172.21.15.122:0/2538567532 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb00b44c0 con 0x7f8bb013f710 2024-03-25T18:19:31.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.781+0000 7f8bb4d30640 10 monclient: _finish_auth 0 2024-03-25T18:19:31.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.781+0000 7f8bb4d30640 10 monclient: _check_auth_tickets 2024-03-25T18:19:31.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.781+0000 7f8bb4d30640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:31.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.781+0000 7f8bb4d30640 1 -- 172.21.15.122:0/2538567532 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f8ba0001690 con 0x7f8bb013f710 2024-03-25T18:19:31.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.781+0000 7f8bb4d30640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:31.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.781+0000 7f8bb4d30640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:31.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.781+0000 7f8bb4d30640 10 monclient: dump: 2024-03-25T18:19:31.783 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:31.783 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:31.783 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:31.783 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:31.784 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:31.784 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:31.784 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:31.784 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:31.784 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:31.784 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:31.784 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.781+0000 7f8bb7f96640 5 monclient: authenticate success, global_id 4365 2024-03-25T18:19:31.784 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.781+0000 7f8bb7f96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:31.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.790+0000 7f8bb4d30640 1 -- 172.21.15.122:0/2538567532 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ba00017f0 con 0x7f8bb013f710 2024-03-25T18:19:31.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.790+0000 7f8bb4d30640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:31.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.790+0000 7f8bb7f96640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:31.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.790+0000 7f8bb7f96640 4 set_mon_vals no callback set 2024-03-25T18:19:31.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.791+0000 7f8bb7f96640 10 monclient: shutdown 2024-03-25T18:19:31.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.791+0000 7f8bb7f96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:31.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.791+0000 7f8bb7f96640 1 -- 172.21.15.122:0/2538567532 >> v1:172.21.15.122:6789/0 conn(0x7f8bb013f710 legacy=0x7f8bb01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.791+0000 7f8bb7f96640 1 -- 172.21.15.122:0/2538567532 shutdown_connections 2024-03-25T18:19:31.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.791+0000 7f8bb7f96640 1 -- 172.21.15.122:0/2538567532 >> 172.21.15.122:0/2538567532 conn(0x7f8bb00b7ea0 msgr2=0x7f8bb00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:31.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.792+0000 7f8bb7f96640 1 -- 172.21.15.122:0/2538567532 shutdown_connections 2024-03-25T18:19:31.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.792+0000 7f8bb7f96640 1 -- 172.21.15.122:0/2538567532 wait complete. 2024-03-25T18:19:31.794 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.793+0000 7f8bb7f96640 10 monclient: build_initial_monmap 2024-03-25T18:19:31.794 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.793+0000 7f8bb7f96640 10 monclient: monmap: 2024-03-25T18:19:31.794 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:19:31.794 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:31.794 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:19:31.794 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:19:31.794 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:31.794 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:19:31.795 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:19:31.795 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:19:31.795 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:19:31.795 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:31.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.793+0000 7f8bb7f96640 1 librados: starting msgr at 2024-03-25T18:19:31.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.793+0000 7f8bb7f96640 1 librados: starting objecter 2024-03-25T18:19:31.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.793+0000 7f8bb7f96640 1 Processor -- start 2024-03-25T18:19:31.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.793+0000 7f8bb7f96640 1 -- start start 2024-03-25T18:19:31.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.793+0000 7f8bb7f96640 1 librados: setting wanted keys 2024-03-25T18:19:31.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.793+0000 7f8bb7f96640 1 librados: calling monclient init 2024-03-25T18:19:31.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.793+0000 7f8bb7f96640 10 monclient: init 2024-03-25T18:19:31.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.794+0000 7f8bb7f96640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:31.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.794+0000 7f8bb7f96640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:31.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.794+0000 7f8bb7f96640 10 monclient(hunting): picked mon.noname-b con 0x7f8bb00a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:19:31.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.794+0000 7f8bb7f96640 10 monclient(hunting): picked mon.noname-c con 0x7f8bb013f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:19:31.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.794+0000 7f8bb7f96640 10 monclient(hunting): picked mon.noname-a con 0x7f8bb01430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:19:31.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.794+0000 7f8bb7f96640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb01db930 con 0x7f8bb01430c0 2024-03-25T18:19:31.797 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.794+0000 7f8bb7f96640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb0139470 con 0x7f8bb00a98f0 2024-03-25T18:19:31.797 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.794+0000 7f8bb7f96640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bb0139750 con 0x7f8bb013f710 2024-03-25T18:19:31.797 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.794+0000 7f8bb7f96640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:31.797 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.794+0000 7f8bb7f96640 10 monclient(hunting): authenticate will time out at 7495.060653s 2024-03-25T18:19:31.799 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.798+0000 7f8bb6533640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f8bb01430c0 0x7f8bb0137990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:34568/0 (socket says 172.21.15.122:34568) 2024-03-25T18:19:31.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.799+0000 7f8bb6533640 1 -- 172.21.15.122:0/1329403580 learned_addr learned my addr 172.21.15.122:0/1329403580 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:19:31.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.799+0000 7f8ba67fc640 10 client.?.objecter ms_handle_connect 0x7f8bb01430c0 2024-03-25T18:19:31.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.799+0000 7f8ba67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:31.802 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.801+0000 7f8ba67fc640 10 client.?.objecter ms_handle_connect 0x7f8bb013f710 2024-03-25T18:19:31.802 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.801+0000 7f8ba67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:31.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.805+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217773008 0 0) 0x7f8bb0139750 con 0x7f8bb013f710 2024-03-25T18:19:31.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.805+0000 7f8ba67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:31.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.805+0000 7f8ba67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:31.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.805+0000 7f8ba67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:31.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.805+0000 7f8ba67fc640 10 monclient(hunting): my global_id is 4346 2024-03-25T18:19:31.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.805+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b94004eb0 con 0x7f8bb013f710 2024-03-25T18:19:31.815 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.814+0000 7f8ba67fc640 10 client.?.objecter ms_handle_connect 0x7f8bb00a98f0 2024-03-25T18:19:31.815 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.814+0000 7f8ba67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:31.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.816+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1855751395 0 0) 0x7f8bb01db930 con 0x7f8bb01430c0 2024-03-25T18:19:31.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.816+0000 7f8ba67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:31.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.816+0000 7f8ba67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:31.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.816+0000 7f8ba67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:31.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.816+0000 7f8ba67fc640 10 monclient(hunting): my global_id is 4368 2024-03-25T18:19:31.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.816+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb0139750 con 0x7f8bb01430c0 2024-03-25T18:19:31.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.816+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3465098731 0 0) 0x7f8b94004eb0 con 0x7f8bb013f710 2024-03-25T18:19:31.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.816+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bb01db930 con 0x7f8bb013f710 2024-03-25T18:19:31.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.816+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f8ba0001ca0 con 0x7f8bb013f710 2024-03-25T18:19:31.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.816+0000 7f8ba67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:31.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.816+0000 7f8ba67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:31.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.816+0000 7f8ba67fc640 10 monclient(hunting): dump: 2024-03-25T18:19:31.818 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:31.818 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:31.819 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:31.819 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:31.819 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:31.819 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:31.819 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:31.819 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:31.819 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:31.819 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:31.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.816+0000 7f8ba67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:31.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.819+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998147798 0 0) 0x7f8bb0139470 con 0x7f8bb00a98f0 2024-03-25T18:19:31.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.819+0000 7f8ba67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:31.821 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.819+0000 7f8ba67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:31.821 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.819+0000 7f8ba67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:31.821 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.819+0000 7f8ba67fc640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:19:31.821 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.819+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b94004eb0 con 0x7f8bb00a98f0 2024-03-25T18:19:31.824 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 37440875 0 0) 0x7f8bb01db930 con 0x7f8bb013f710 2024-03-25T18:19:31.824 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 >> v1:172.21.15.204:6789/0 conn(0x7f8bb00a98f0 legacy=0x7f8bb00b0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.824 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 >> v1:172.21.15.122:6789/0 conn(0x7f8bb01430c0 legacy=0x7f8bb0137990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.824 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8ba67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:31.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8ba67fc640 1 monclient: found mon.c 2024-03-25T18:19:31.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8ba67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:31.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8ba67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:19:31.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb00b0a60 con 0x7f8bb013f710 2024-03-25T18:19:31.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8ba67fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:31.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8ba67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:31.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8ba67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:31.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8bb7f96640 5 monclient: authenticate success, global_id 4346 2024-03-25T18:19:31.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8bb7f96640 10 monclient: _renew_subs 2024-03-25T18:19:31.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8bb7f96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:19:31.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8bb7f96640 1 -- 172.21.15.122:0/1329403580 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb00b0cb0 con 0x7f8bb013f710 2024-03-25T18:19:31.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8bb7f96640 10 client.4346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:31.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8bb7f96640 10 monclient: _renew_subs 2024-03-25T18:19:31.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8bb7f96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:19:31.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8bb7f96640 1 -- 172.21.15.122:0/1329403580 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb01db560 con 0x7f8bb013f710 2024-03-25T18:19:31.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.823+0000 7f8bb7f96640 1 librados: init done 2024-03-25T18:19:31.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.824+0000 7f8bb7f96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:31.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.824+0000 7f8bb7f96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:31.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.824+0000 7f8bb7f96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:19:31.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:31.824+0000 7f8bb7f96640 1 -- 172.21.15.122:0/1329403580 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb00627b0 con 0x7f8bb013f710 2024-03-25T18:19:32.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.119+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ba0004a90 con 0x7f8bb013f710 2024-03-25T18:19:32.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.119+0000 7f8ba67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:32.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.119+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f8ba0004da0 con 0x7f8bb013f710 2024-03-25T18:19:32.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.119+0000 7f8ba67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:32.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.119+0000 7f8ba67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:32.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.119+0000 7f8ba67fc640 10 monclient: dump: 2024-03-25T18:19:32.121 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:32.121 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:32.121 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:32.121 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:32.121 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:32.121 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:32.121 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:32.121 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:32.121 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:32.121 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:32.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.119+0000 7f8ba7fff640 4 set_mon_vals no callback set 2024-03-25T18:19:32.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.120+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f8ba001ad70 con 0x7f8bb013f710 2024-03-25T18:19:32.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.121+0000 7f8bb5d32640 10 monclient: get_auth_request con 0x7f8b9406de00 auth_method 0 2024-03-25T18:19:32.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.121+0000 7f8ba67fc640 10 client.4346.objecter ms_handle_connect 0x7f8b9406de00 2024-03-25T18:19:32.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.126+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f8ba00548f0 con 0x7f8bb013f710 2024-03-25T18:19:32.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.126+0000 7f8ba67fc640 10 client.4346.objecter ms_dispatch 0x7f8bb0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:32.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.126+0000 7f8ba67fc640 3 client.4346.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:32.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.126+0000 7f8ba67fc640 3 client.4346.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:32.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.126+0000 7f8ba67fc640 20 client.4346.objecter dump_active .. 0 homeless 2024-03-25T18:19:32.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.126+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ba0050400 con 0x7f8bb013f710 2024-03-25T18:19:32.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.126+0000 7f8ba67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:32.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.126+0000 7f8ba67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:32.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.376+0000 7f8bb7f96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:32.378 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.376+0000 7f8bb7f96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:32.378 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.376+0000 7f8bb7f96640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:19:32.378 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.376+0000 7f8bb7f96640 1 -- 172.21.15.122:0/1329403580 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8bb01376b0 con 0x7f8bb013f710 2024-03-25T18:19:32.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.405+0000 7f8ba67fc640 1 -- 172.21.15.122:0/1329403580 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f8bb01376b0 con 0x7f8bb013f710 2024-03-25T18:19:32.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.405+0000 7f8ba67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:32.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.405+0000 7f8ba67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:32.406 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:19:32.406 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:32.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.408+0000 7f8bb7f96640 10 librados: watch_flush enter 2024-03-25T18:19:32.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.408+0000 7f8bb7f96640 10 librados: watch_flush exit 2024-03-25T18:19:32.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.408+0000 7f8bb7f96640 20 client.4346.objecter shutdown clearing up homeless session... 2024-03-25T18:19:32.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.408+0000 7f8bb7f96640 10 client.4346.objecter successfully canceled tick 2024-03-25T18:19:32.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.408+0000 7f8bb7f96640 1 -- 172.21.15.122:0/1329403580 >> v1:172.21.15.204:6800/676228005 conn(0x7f8b9406de00 legacy=0x7f8b940702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:32.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.408+0000 7f8bb7f96640 10 monclient: shutdown 2024-03-25T18:19:32.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.408+0000 7f8bb7f96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:32.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.408+0000 7f8bb7f96640 1 -- 172.21.15.122:0/1329403580 >> v1:172.21.15.122:6790/0 conn(0x7f8bb013f710 legacy=0x7f8bb01dac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:32.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.408+0000 7f8bb7f96640 1 -- 172.21.15.122:0/1329403580 shutdown_connections 2024-03-25T18:19:32.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.408+0000 7f8bb7f96640 1 -- 172.21.15.122:0/1329403580 >> 172.21.15.122:0/1329403580 conn(0x7f8bb00b7ea0 msgr2=0x7f8bb0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:32.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.408+0000 7f8bb7f96640 1 -- 172.21.15.122:0/1329403580 shutdown_connections 2024-03-25T18:19:32.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.408+0000 7f8bb7f96640 1 -- 172.21.15.122:0/1329403580 wait complete. 2024-03-25T18:19:32.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:32.408+0000 7f8bb7f96640 1 librados: shutdown 2024-03-25T18:19:32.421 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:39.227 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:39.346 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.344+0000 7fab046f4640 10 monclient: get_monmap_and_config 2024-03-25T18:19:39.346 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.344+0000 7fab046f4640 10 monclient: build_initial_monmap 2024-03-25T18:19:39.346 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.344+0000 7fab046f4640 10 monclient: monmap: 2024-03-25T18:19:39.346 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:19:39.346 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:39.346 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:19:39.345819+0000 2024-03-25T18:19:39.347 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:19:39.345819+0000 2024-03-25T18:19:39.347 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:39.347 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:39.347 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:19:39.347 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:19:39.347 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:19:39.347 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:39.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.345+0000 7fab046f4640 1 Processor -- start 2024-03-25T18:19:39.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.345+0000 7fab046f4640 1 -- start start 2024-03-25T18:19:39.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.345+0000 7fab046f4640 10 monclient: init 2024-03-25T18:19:39.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.346+0000 7fab046f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:39.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.346+0000 7fab046f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.346+0000 7fab046f4640 10 monclient(hunting): picked mon.noname-a con 0x7faafc0a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.346+0000 7fab046f4640 10 monclient(hunting): picked mon.noname-b con 0x7faafc13f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.346+0000 7fab046f4640 10 monclient(hunting): picked mon.noname-c con 0x7faafc1430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.346+0000 7fab046f4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faafc142520 con 0x7faafc0a98f0 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.346+0000 7fab046f4640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faafc0b28a0 con 0x7faafc1430c0 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.346+0000 7fab046f4640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faafc0b2a30 con 0x7faafc13f710 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.346+0000 7fab046f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:39.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.346+0000 7fab046f4640 10 monclient(hunting): authenticate will time out at 7502.612570s 2024-03-25T18:19:39.352 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.351+0000 7fab02490640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7faafc0a98f0 0x7faafc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:39268/0 (socket says 172.21.15.122:39268) 2024-03-25T18:19:39.352 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.351+0000 7fab02490640 1 -- 172.21.15.122:0/217972822 learned_addr learned my addr 172.21.15.122:0/217972822 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:19:39.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.360+0000 7fab0148e640 1 -- 172.21.15.122:0/217972822 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1202569800 0 0) 0x7faafc142520 con 0x7faafc0a98f0 2024-03-25T18:19:39.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.360+0000 7fab0148e640 10 monclient(hunting): _negotiate 2024-03-25T18:19:39.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.360+0000 7fab0148e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:39.362 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.360+0000 7fab0148e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:39.362 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.360+0000 7fab0148e640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:19:39.362 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.360+0000 7fab0148e640 1 -- 172.21.15.122:0/217972822 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faae8003520 con 0x7faafc0a98f0 2024-03-25T18:19:39.367 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab0148e640 1 -- 172.21.15.122:0/217972822 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2959071597 0 0) 0x7faae8003520 con 0x7faafc0a98f0 2024-03-25T18:19:39.367 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab0148e640 1 -- 172.21.15.122:0/217972822 >> v1:172.21.15.122:6790/0 conn(0x7faafc13f710 legacy=0x7faafc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab0148e640 1 -- 172.21.15.122:0/217972822 >> v1:172.21.15.204:6789/0 conn(0x7faafc1430c0 legacy=0x7faafc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab0148e640 10 monclient: _finish_hunting 0 2024-03-25T18:19:39.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab0148e640 1 monclient: found mon.noname-a 2024-03-25T18:19:39.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab0148e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:39.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab0148e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:19:39.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab0148e640 1 -- 172.21.15.122:0/217972822 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faafc0b44c0 con 0x7faafc0a98f0 2024-03-25T18:19:39.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab0148e640 10 monclient: _finish_auth 0 2024-03-25T18:19:39.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab0148e640 10 monclient: _check_auth_tickets 2024-03-25T18:19:39.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab0148e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:39.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab0148e640 1 -- 172.21.15.122:0/217972822 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7faaf8001690 con 0x7faafc0a98f0 2024-03-25T18:19:39.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab046f4640 5 monclient: authenticate success, global_id 4371 2024-03-25T18:19:39.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab046f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:39.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab0148e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:39.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab0148e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:39.369 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab0148e640 10 monclient: dump: 2024-03-25T18:19:39.369 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:39.369 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:39.369 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:39.369 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:39.369 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:39.369 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:39.369 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:39.369 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:39.369 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:39.369 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:39.369 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.366+0000 7fab046f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:39.378 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.377+0000 7fab0148e640 1 -- 172.21.15.122:0/217972822 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faaf8001870 con 0x7faafc0a98f0 2024-03-25T18:19:39.378 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.377+0000 7fab0148e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:39.378 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.377+0000 7fab0148e640 1 -- 172.21.15.122:0/217972822 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7faaf8001c80 con 0x7faafc0a98f0 2024-03-25T18:19:39.378 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.377+0000 7fab046f4640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:39.378 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.377+0000 7fab046f4640 4 set_mon_vals no callback set 2024-03-25T18:19:39.378 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.377+0000 7fab0148e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:39.378 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.377+0000 7fab0148e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:39.378 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.377+0000 7fab0148e640 10 monclient: dump: 2024-03-25T18:19:39.378 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:39.379 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:39.379 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:39.379 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:39.379 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:39.379 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:39.379 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:39.379 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:39.379 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:39.379 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:39.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.378+0000 7fab046f4640 10 monclient: shutdown 2024-03-25T18:19:39.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.378+0000 7fab046f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:39.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.378+0000 7fab046f4640 1 -- 172.21.15.122:0/217972822 >> v1:172.21.15.122:6789/0 conn(0x7faafc0a98f0 legacy=0x7faafc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.378+0000 7fab046f4640 1 -- 172.21.15.122:0/217972822 shutdown_connections 2024-03-25T18:19:39.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.378+0000 7fab046f4640 1 -- 172.21.15.122:0/217972822 >> 172.21.15.122:0/217972822 conn(0x7faafc0b7ea0 msgr2=0x7faafc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:39.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.378+0000 7fab046f4640 1 -- 172.21.15.122:0/217972822 shutdown_connections 2024-03-25T18:19:39.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.378+0000 7fab046f4640 1 -- 172.21.15.122:0/217972822 wait complete. 2024-03-25T18:19:39.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.379+0000 7fab046f4640 10 monclient: build_initial_monmap 2024-03-25T18:19:39.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.379+0000 7fab046f4640 10 monclient: monmap: 2024-03-25T18:19:39.381 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:19:39.381 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:39.381 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:19:39.381 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:19:39.381 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:39.381 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:19:39.381 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:19:39.381 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:19:39.381 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:19:39.381 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:39.383 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.379+0000 7fab046f4640 1 librados: starting msgr at 2024-03-25T18:19:39.383 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.379+0000 7fab046f4640 1 librados: starting objecter 2024-03-25T18:19:39.383 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.379+0000 7fab046f4640 1 Processor -- start 2024-03-25T18:19:39.383 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.379+0000 7fab046f4640 1 -- start start 2024-03-25T18:19:39.383 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.379+0000 7fab046f4640 1 librados: setting wanted keys 2024-03-25T18:19:39.383 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.379+0000 7fab046f4640 1 librados: calling monclient init 2024-03-25T18:19:39.383 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.379+0000 7fab046f4640 10 monclient: init 2024-03-25T18:19:39.383 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.380+0000 7fab046f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:39.383 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.380+0000 7fab046f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:39.383 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.380+0000 7fab046f4640 10 monclient(hunting): picked mon.noname-a con 0x7faafc0a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:19:39.383 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.380+0000 7fab046f4640 10 monclient(hunting): picked mon.noname-c con 0x7faafc13f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:19:39.383 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.380+0000 7fab046f4640 10 monclient(hunting): picked mon.noname-b con 0x7faafc1430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:19:39.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.380+0000 7fab046f4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faafc0b1750 con 0x7faafc0a98f0 2024-03-25T18:19:39.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.380+0000 7fab046f4640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faafc1399e0 con 0x7faafc1430c0 2024-03-25T18:19:39.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.380+0000 7fab046f4640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faafc0ab950 con 0x7faafc13f710 2024-03-25T18:19:39.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.380+0000 7fab046f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:39.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.380+0000 7fab046f4640 10 monclient(hunting): authenticate will time out at 7502.646934s 2024-03-25T18:19:39.388 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.388+0000 7fab02c91640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7faafc1430c0 0x7faafc1e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:44758/0 (socket says 172.21.15.122:44758) 2024-03-25T18:19:39.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.388+0000 7fab02c91640 1 -- 172.21.15.122:0/4049853477 learned_addr learned my addr 172.21.15.122:0/4049853477 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:19:39.390 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.389+0000 7faaf2ffd640 10 client.?.objecter ms_handle_connect 0x7faafc0a98f0 2024-03-25T18:19:39.390 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.389+0000 7faaf2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:39.390 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.389+0000 7faaf2ffd640 10 client.?.objecter ms_handle_connect 0x7faafc1430c0 2024-03-25T18:19:39.390 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.389+0000 7faaf2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:39.398 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.397+0000 7faaf2ffd640 1 -- 172.21.15.122:0/4049853477 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2430659612 0 0) 0x7faafc0b1750 con 0x7faafc0a98f0 2024-03-25T18:19:39.398 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.397+0000 7faaf2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:39.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.397+0000 7faaf2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:39.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.397+0000 7faaf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:39.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.397+0000 7faaf2ffd640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:19:39.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.397+0000 7faaf2ffd640 1 -- 172.21.15.122:0/4049853477 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faae0004eb0 con 0x7faafc0a98f0 2024-03-25T18:19:39.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.397+0000 7faaf2ffd640 1 -- 172.21.15.122:0/4049853477 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160614482 0 0) 0x7faafc1399e0 con 0x7faafc1430c0 2024-03-25T18:19:39.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.397+0000 7faaf2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:39.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.397+0000 7faaf2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:39.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.397+0000 7faaf2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:39.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.397+0000 7faaf2ffd640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:19:39.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.397+0000 7faaf2ffd640 1 -- 172.21.15.122:0/4049853477 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faafc0b1750 con 0x7faafc1430c0 2024-03-25T18:19:39.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.406+0000 7faaf2ffd640 10 client.?.objecter ms_handle_connect 0x7faafc13f710 2024-03-25T18:19:39.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.406+0000 7faaf2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:39.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.408+0000 7faaf2ffd640 1 -- 172.21.15.122:0/4049853477 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1666339241 0 0) 0x7faafc0b1750 con 0x7faafc1430c0 2024-03-25T18:19:39.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.408+0000 7faaf2ffd640 1 -- 172.21.15.122:0/4049853477 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faafc1399e0 con 0x7faafc1430c0 2024-03-25T18:19:39.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.408+0000 7faaf2ffd640 1 -- 172.21.15.122:0/4049853477 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7faaf4001c50 con 0x7faafc1430c0 2024-03-25T18:19:39.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.408+0000 7faaf2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:39.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.408+0000 7faaf2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:39.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.408+0000 7faaf2ffd640 10 monclient(hunting): dump: 2024-03-25T18:19:39.409 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:39.409 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:39.409 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:39.409 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:39.409 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:39.410 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:39.410 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:39.410 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:39.410 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:39.410 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:39.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.408+0000 7faaf2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:39.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7faaf2ffd640 1 -- 172.21.15.122:0/4049853477 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2596325865 0 0) 0x7faafc1399e0 con 0x7faafc1430c0 2024-03-25T18:19:39.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7faaf2ffd640 1 -- 172.21.15.122:0/4049853477 >> v1:172.21.15.122:6790/0 conn(0x7faafc13f710 legacy=0x7faafc136350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7faaf2ffd640 1 -- 172.21.15.122:0/4049853477 >> v1:172.21.15.122:6789/0 conn(0x7faafc0a98f0 legacy=0x7faafc0ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7faaf2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:39.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7faaf2ffd640 1 monclient: found mon.b 2024-03-25T18:19:39.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7faaf2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:39.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7faaf2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:19:39.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7faaf2ffd640 1 -- 172.21.15.122:0/4049853477 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faafc0abc30 con 0x7faafc1430c0 2024-03-25T18:19:39.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7faaf2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:39.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7faaf2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:39.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7faaf2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:39.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7fab046f4640 5 monclient: authenticate success, global_id 4372 2024-03-25T18:19:39.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7fab046f4640 10 monclient: _renew_subs 2024-03-25T18:19:39.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7fab046f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:19:39.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7fab046f4640 1 -- 172.21.15.122:0/4049853477 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faafc138ea0 con 0x7faafc1430c0 2024-03-25T18:19:39.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7fab046f4640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:39.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7fab046f4640 10 monclient: _renew_subs 2024-03-25T18:19:39.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7fab046f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:19:39.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7fab046f4640 1 -- 172.21.15.122:0/4049853477 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faafc1387c0 con 0x7faafc1430c0 2024-03-25T18:19:39.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.412+0000 7fab046f4640 1 librados: init done 2024-03-25T18:19:39.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.413+0000 7fab046f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:39.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.413+0000 7fab046f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:39.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.413+0000 7fab046f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:19:39.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.413+0000 7fab046f4640 1 -- 172.21.15.122:0/4049853477 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faafc0627b0 con 0x7faafc1430c0 2024-03-25T18:19:39.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.690+0000 7faaf2ffd640 1 -- 172.21.15.122:0/4049853477 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faaf4003990 con 0x7faafc1430c0 2024-03-25T18:19:39.692 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.690+0000 7faaf2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:39.692 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.690+0000 7fab00c8d640 4 set_mon_vals no callback set 2024-03-25T18:19:39.692 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.691+0000 7faaf2ffd640 1 -- 172.21.15.122:0/4049853477 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7faaf4003ce0 con 0x7faafc1430c0 2024-03-25T18:19:39.692 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.691+0000 7faaf2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:39.692 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.691+0000 7faaf2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:39.692 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.691+0000 7faaf2ffd640 10 monclient: dump: 2024-03-25T18:19:39.692 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:39.693 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:39.693 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:39.693 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:39.693 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:39.693 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:39.693 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:39.693 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:39.693 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:39.693 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:39.693 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.691+0000 7faaf2ffd640 1 -- 172.21.15.122:0/4049853477 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7faaf4003f00 con 0x7faafc1430c0 2024-03-25T18:19:39.694 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.692+0000 7fab02490640 10 monclient: get_auth_request con 0x7faae006de20 auth_method 0 2024-03-25T18:19:39.694 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.692+0000 7faaf2ffd640 10 client.4372.objecter ms_handle_connect 0x7faae006de20 2024-03-25T18:19:39.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.696+0000 7faaf2ffd640 1 -- 172.21.15.122:0/4049853477 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7faaf4054af0 con 0x7faafc1430c0 2024-03-25T18:19:39.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.696+0000 7faaf2ffd640 10 client.4372.objecter ms_dispatch 0x7faafc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:39.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.696+0000 7faaf2ffd640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:39.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.696+0000 7faaf2ffd640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:39.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.696+0000 7faaf2ffd640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-25T18:19:39.698 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.697+0000 7faaf2ffd640 1 -- 172.21.15.122:0/4049853477 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faaf4050600 con 0x7faafc1430c0 2024-03-25T18:19:39.698 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.697+0000 7faaf2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:39.698 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.697+0000 7faaf2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:39.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.944+0000 7fab046f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:39.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.944+0000 7fab046f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:39.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.944+0000 7fab046f4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:19:39.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.944+0000 7fab046f4640 1 -- 172.21.15.122:0/4049853477 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faafc139030 con 0x7faafc1430c0 2024-03-25T18:19:39.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.969+0000 7faaf2ffd640 1 -- 172.21.15.122:0/4049853477 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7faaf4052530 con 0x7faafc1430c0 2024-03-25T18:19:39.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.969+0000 7faaf2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:39.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.969+0000 7faaf2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:39.971 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:19:39.971 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:39.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.971+0000 7fab046f4640 10 librados: watch_flush enter 2024-03-25T18:19:39.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.972+0000 7fab046f4640 10 librados: watch_flush exit 2024-03-25T18:19:39.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.972+0000 7fab046f4640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-25T18:19:39.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.972+0000 7fab046f4640 10 client.4372.objecter successfully canceled tick 2024-03-25T18:19:39.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.972+0000 7fab046f4640 1 -- 172.21.15.122:0/4049853477 >> v1:172.21.15.204:6800/676228005 conn(0x7faae006de20 legacy=0x7faae00702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.972+0000 7fab046f4640 10 monclient: shutdown 2024-03-25T18:19:39.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.972+0000 7fab046f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:39.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.972+0000 7fab046f4640 1 -- 172.21.15.122:0/4049853477 >> v1:172.21.15.204:6789/0 conn(0x7faafc1430c0 legacy=0x7faafc1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.972+0000 7fab046f4640 1 -- 172.21.15.122:0/4049853477 shutdown_connections 2024-03-25T18:19:39.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.972+0000 7fab046f4640 1 -- 172.21.15.122:0/4049853477 >> 172.21.15.122:0/4049853477 conn(0x7faafc0b7ea0 msgr2=0x7faafc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:39.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.972+0000 7fab046f4640 1 -- 172.21.15.122:0/4049853477 shutdown_connections 2024-03-25T18:19:39.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.972+0000 7fab046f4640 1 -- 172.21.15.122:0/4049853477 wait complete. 2024-03-25T18:19:39.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:39.972+0000 7fab046f4640 1 librados: shutdown 2024-03-25T18:19:39.985 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:45.897 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:03:39.640 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:45.901 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:03:39.643 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:45.927 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:03:39.670 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:45.955 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:03:39.698 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:45.961 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:03:39.706 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:45.972 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:03:39.715 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:45.975 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:03:39.721 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:46.001 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:03:39.747 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:46.010 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:03:39.753 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:46.792 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:46.905 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.904+0000 7fdcca7ed640 10 monclient: get_monmap_and_config 2024-03-25T18:19:46.906 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.904+0000 7fdcca7ed640 10 monclient: build_initial_monmap 2024-03-25T18:19:46.906 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.904+0000 7fdcca7ed640 10 monclient: monmap: 2024-03-25T18:19:46.906 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:19:46.906 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:46.906 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:19:46.906079+0000 2024-03-25T18:19:46.906 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:19:46.906079+0000 2024-03-25T18:19:46.906 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:46.906 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:46.906 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:19:46.906 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:19:46.906 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:19:46.906 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:46.907 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.905+0000 7fdcca7ed640 1 Processor -- start 2024-03-25T18:19:46.907 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.905+0000 7fdcca7ed640 1 -- start start 2024-03-25T18:19:46.907 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.905+0000 7fdcca7ed640 10 monclient: init 2024-03-25T18:19:46.907 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.906+0000 7fdcca7ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:46.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.906+0000 7fdcca7ed640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:46.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.906+0000 7fdcca7ed640 10 monclient(hunting): picked mon.noname-c con 0x7fdcc40b4d60 addr v1:172.21.15.204:6789/0 2024-03-25T18:19:46.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.906+0000 7fdcca7ed640 10 monclient(hunting): picked mon.noname-b con 0x7fdcc40b1730 addr v1:172.21.15.122:6790/0 2024-03-25T18:19:46.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.906+0000 7fdcca7ed640 10 monclient(hunting): picked mon.noname-a con 0x7fdcc414be90 addr v1:172.21.15.122:6789/0 2024-03-25T18:19:46.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.906+0000 7fdcca7ed640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc40af940 con 0x7fdcc414be90 2024-03-25T18:19:46.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.906+0000 7fdcca7ed640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc40b0a70 con 0x7fdcc40b4d60 2024-03-25T18:19:46.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.906+0000 7fdcca7ed640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc40b0d50 con 0x7fdcc40b1730 2024-03-25T18:19:46.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.906+0000 7fdcca7ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:46.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.906+0000 7fdcca7ed640 10 monclient(hunting): authenticate will time out at 7510.172361s 2024-03-25T18:19:46.941 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.940+0000 7fdcc3fff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fdcc40b4d60 0x7fdcc413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:42116/0 (socket says 172.21.15.122:42116) 2024-03-25T18:19:46.941 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.940+0000 7fdcc3fff640 1 -- 172.21.15.122:0/1417283265 learned_addr learned my addr 172.21.15.122:0/1417283265 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:19:46.956 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.955+0000 7fdcc2ffd640 1 -- 172.21.15.122:0/1417283265 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4042615742 0 0) 0x7fdcc40b0a70 con 0x7fdcc40b4d60 2024-03-25T18:19:46.956 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.955+0000 7fdcc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:46.957 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.955+0000 7fdcc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:46.957 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.955+0000 7fdcc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:46.957 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.955+0000 7fdcc2ffd640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:19:46.957 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.955+0000 7fdcc2ffd640 1 -- 172.21.15.122:0/1417283265 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcac003520 con 0x7fdcc40b4d60 2024-03-25T18:19:46.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.960+0000 7fdcc2ffd640 1 -- 172.21.15.122:0/1417283265 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605930665 0 0) 0x7fdcc40b0d50 con 0x7fdcc40b1730 2024-03-25T18:19:46.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.960+0000 7fdcc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:46.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.960+0000 7fdcc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:46.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.960+0000 7fdcc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:46.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.960+0000 7fdcc2ffd640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:19:46.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.960+0000 7fdcc2ffd640 1 -- 172.21.15.122:0/1417283265 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc40b0a70 con 0x7fdcc40b1730 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.972+0000 7fdcc2ffd640 1 -- 172.21.15.122:0/1417283265 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217554055 0 0) 0x7fdcc40af940 con 0x7fdcc414be90 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.972+0000 7fdcc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.972+0000 7fdcc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.972+0000 7fdcc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:46.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.972+0000 7fdcc2ffd640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:19:46.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.972+0000 7fdcc2ffd640 1 -- 172.21.15.122:0/1417283265 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcc40b0d50 con 0x7fdcc414be90 2024-03-25T18:19:46.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.978+0000 7fdcc2ffd640 1 -- 172.21.15.122:0/1417283265 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 252321879 0 0) 0x7fdcac003520 con 0x7fdcc40b4d60 2024-03-25T18:19:46.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.978+0000 7fdcc2ffd640 1 -- 172.21.15.122:0/1417283265 >> v1:172.21.15.122:6790/0 conn(0x7fdcc40b1730 legacy=0x7fdcc40b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.978+0000 7fdcc2ffd640 1 -- 172.21.15.122:0/1417283265 >> v1:172.21.15.122:6789/0 conn(0x7fdcc414be90 legacy=0x7fdcc414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.978+0000 7fdcc2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:46.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.978+0000 7fdcc2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:19:46.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.978+0000 7fdcc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:46.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.979+0000 7fdcc2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:19:46.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.979+0000 7fdcc2ffd640 1 -- 172.21.15.122:0/1417283265 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc40b1030 con 0x7fdcc40b4d60 2024-03-25T18:19:46.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.979+0000 7fdcc2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:46.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.979+0000 7fdcc2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.979+0000 7fdcc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.979+0000 7fdcc2ffd640 1 -- 172.21.15.122:0/1417283265 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fdcb4001690 con 0x7fdcc40b4d60 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.979+0000 7fdcc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.979+0000 7fdcc2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.979+0000 7fdcc2ffd640 10 monclient: dump: 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.979+0000 7fdcca7ed640 5 monclient: authenticate success, global_id 4375 2024-03-25T18:19:46.981 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.979+0000 7fdcca7ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:46.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.985+0000 7fdcc2ffd640 1 -- 172.21.15.122:0/1417283265 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcb40017f0 con 0x7fdcc40b4d60 2024-03-25T18:19:46.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.985+0000 7fdcc2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:46.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.985+0000 7fdcc2ffd640 1 -- 172.21.15.122:0/1417283265 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fdcb4001c00 con 0x7fdcc40b4d60 2024-03-25T18:19:46.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.985+0000 7fdcc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:46.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.985+0000 7fdcc2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:46.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.985+0000 7fdcc2ffd640 10 monclient: dump: 2024-03-25T18:19:46.986 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.985+0000 7fdcca7ed640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:46.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.985+0000 7fdcca7ed640 4 set_mon_vals no callback set 2024-03-25T18:19:46.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.986+0000 7fdcca7ed640 10 monclient: shutdown 2024-03-25T18:19:46.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.986+0000 7fdcca7ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:46.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.986+0000 7fdcca7ed640 1 -- 172.21.15.122:0/1417283265 >> v1:172.21.15.204:6789/0 conn(0x7fdcc40b4d60 legacy=0x7fdcc413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:46.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.986+0000 7fdcca7ed640 1 -- 172.21.15.122:0/1417283265 shutdown_connections 2024-03-25T18:19:46.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.986+0000 7fdcca7ed640 1 -- 172.21.15.122:0/1417283265 >> 172.21.15.122:0/1417283265 conn(0x7fdcc40617a0 msgr2=0x7fdcc4061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:46.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.986+0000 7fdcca7ed640 1 -- 172.21.15.122:0/1417283265 shutdown_connections 2024-03-25T18:19:46.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.986+0000 7fdcca7ed640 1 -- 172.21.15.122:0/1417283265 wait complete. 2024-03-25T18:19:46.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.987+0000 7fdcca7ed640 10 monclient: build_initial_monmap 2024-03-25T18:19:46.989 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.987+0000 7fdcca7ed640 10 monclient: monmap: 2024-03-25T18:19:46.989 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:19:46.989 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:46.989 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:19:46.989 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:19:46.989 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:46.989 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:19:46.990 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:19:46.990 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:19:46.990 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:19:46.990 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:46.990 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.987+0000 7fdcca7ed640 1 librados: starting msgr at 2024-03-25T18:19:46.990 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.987+0000 7fdcca7ed640 1 librados: starting objecter 2024-03-25T18:19:46.990 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.987+0000 7fdcca7ed640 1 Processor -- start 2024-03-25T18:19:46.990 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.987+0000 7fdcca7ed640 1 -- start start 2024-03-25T18:19:46.990 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.987+0000 7fdcca7ed640 1 librados: setting wanted keys 2024-03-25T18:19:46.990 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.987+0000 7fdcca7ed640 1 librados: calling monclient init 2024-03-25T18:19:46.990 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.987+0000 7fdcca7ed640 10 monclient: init 2024-03-25T18:19:46.991 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.988+0000 7fdcca7ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:46.991 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.988+0000 7fdcca7ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:46.991 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.988+0000 7fdcca7ed640 10 monclient(hunting): picked mon.noname-c con 0x7fdcc40b1730 addr v1:172.21.15.122:6790/0 2024-03-25T18:19:46.991 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.988+0000 7fdcca7ed640 10 monclient(hunting): picked mon.noname-b con 0x7fdcc40b4d60 addr v1:172.21.15.204:6789/0 2024-03-25T18:19:46.991 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.988+0000 7fdcca7ed640 10 monclient(hunting): picked mon.noname-a con 0x7fdcc414be90 addr v1:172.21.15.122:6789/0 2024-03-25T18:19:46.991 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.988+0000 7fdcca7ed640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc4143350 con 0x7fdcc414be90 2024-03-25T18:19:46.991 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.988+0000 7fdcca7ed640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc4142110 con 0x7fdcc40b4d60 2024-03-25T18:19:46.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.988+0000 7fdcca7ed640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcc41408f0 con 0x7fdcc40b1730 2024-03-25T18:19:46.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.988+0000 7fdcca7ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:46.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.988+0000 7fdcca7ed640 10 monclient(hunting): authenticate will time out at 7510.254719s 2024-03-25T18:19:46.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.991+0000 7fdcc8d8a640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fdcc414be90 0x7fdcc4142c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:40812/0 (socket says 172.21.15.122:40812) 2024-03-25T18:19:46.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.991+0000 7fdcc8d8a640 1 -- 172.21.15.122:0/3394619778 learned_addr learned my addr 172.21.15.122:0/3394619778 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:19:46.994 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.993+0000 7fdcc0ff9640 10 client.?.objecter ms_handle_connect 0x7fdcc414be90 2024-03-25T18:19:46.994 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.993+0000 7fdcc0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:46.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.996+0000 7fdcc0ff9640 10 client.?.objecter ms_handle_connect 0x7fdcc40b4d60 2024-03-25T18:19:46.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:46.996+0000 7fdcc0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:47.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.002+0000 7fdcc0ff9640 1 -- 172.21.15.122:0/3394619778 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063195510 0 0) 0x7fdcc4143350 con 0x7fdcc414be90 2024-03-25T18:19:47.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.002+0000 7fdcc0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.002+0000 7fdcc0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.002+0000 7fdcc0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.002+0000 7fdcc0ff9640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:19:47.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.002+0000 7fdcc0ff9640 1 -- 172.21.15.122:0/3394619778 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcb0004eb0 con 0x7fdcc414be90 2024-03-25T18:19:47.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.007+0000 7fdcc0ff9640 10 client.?.objecter ms_handle_connect 0x7fdcc40b1730 2024-03-25T18:19:47.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.007+0000 7fdcc0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:47.071 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.070+0000 7fdcc0ff9640 1 -- 172.21.15.122:0/3394619778 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4074715778 0 0) 0x7fdcb0004eb0 con 0x7fdcc414be90 2024-03-25T18:19:47.071 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.070+0000 7fdcc0ff9640 1 -- 172.21.15.122:0/3394619778 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdcc4143350 con 0x7fdcc414be90 2024-03-25T18:19:47.071 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.070+0000 7fdcc0ff9640 1 -- 172.21.15.122:0/3394619778 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fdcbc001d00 con 0x7fdcc414be90 2024-03-25T18:19:47.072 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.070+0000 7fdcc0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.072 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.070+0000 7fdcc0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:47.072 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.070+0000 7fdcc0ff9640 10 monclient(hunting): dump: 2024-03-25T18:19:47.072 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:47.072 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:47.072 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:47.072 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:47.072 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.072 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:47.072 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:47.072 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:47.072 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:47.072 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:47.072 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.070+0000 7fdcc0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:47.075 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.074+0000 7fdcc0ff9640 1 -- 172.21.15.122:0/3394619778 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473864434 0 0) 0x7fdcc41408f0 con 0x7fdcc40b1730 2024-03-25T18:19:47.075 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.074+0000 7fdcc0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.075 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.074+0000 7fdcc0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.075 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.074+0000 7fdcc0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.075 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.074+0000 7fdcc0ff9640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:19:47.075 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.074+0000 7fdcc0ff9640 1 -- 172.21.15.122:0/3394619778 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcb0004eb0 con 0x7fdcc40b1730 2024-03-25T18:19:47.078 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcc0ff9640 1 -- 172.21.15.122:0/3394619778 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1673565387 0 0) 0x7fdcc4143350 con 0x7fdcc414be90 2024-03-25T18:19:47.078 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcc0ff9640 1 -- 172.21.15.122:0/3394619778 >> v1:172.21.15.122:6790/0 conn(0x7fdcc40b1730 legacy=0x7fdcc4134160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.078 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcc0ff9640 1 -- 172.21.15.122:0/3394619778 >> v1:172.21.15.204:6789/0 conn(0x7fdcc40b4d60 legacy=0x7fdcc4142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcc0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:47.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcc0ff9640 1 monclient: found mon.a 2024-03-25T18:19:47.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcc0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:47.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcc0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:47.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcc0ff9640 1 -- 172.21.15.122:0/3394619778 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc4140eb0 con 0x7fdcc414be90 2024-03-25T18:19:47.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcc0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:47.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcc0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:47.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcc0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:47.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcca7ed640 5 monclient: authenticate success, global_id 4380 2024-03-25T18:19:47.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcca7ed640 10 monclient: _renew_subs 2024-03-25T18:19:47.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcca7ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:47.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcca7ed640 1 -- 172.21.15.122:0/3394619778 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcc4141100 con 0x7fdcc414be90 2024-03-25T18:19:47.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcca7ed640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:47.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcca7ed640 10 monclient: _renew_subs 2024-03-25T18:19:47.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcca7ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:47.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcca7ed640 1 -- 172.21.15.122:0/3394619778 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcc4134b00 con 0x7fdcc414be90 2024-03-25T18:19:47.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.077+0000 7fdcca7ed640 1 librados: init done 2024-03-25T18:19:47.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.078+0000 7fdcca7ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:47.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.078+0000 7fdcca7ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:47.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.078+0000 7fdcca7ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:47.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.078+0000 7fdcca7ed640 1 -- 172.21.15.122:0/3394619778 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcc41401f0 con 0x7fdcc414be90 2024-03-25T18:19:47.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.395+0000 7fdcc0ff9640 1 -- 172.21.15.122:0/3394619778 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdcbc003990 con 0x7fdcc414be90 2024-03-25T18:19:47.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.395+0000 7fdcc0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:47.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.395+0000 7fdcc0ff9640 1 -- 172.21.15.122:0/3394619778 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fdcbc003de0 con 0x7fdcc414be90 2024-03-25T18:19:47.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.395+0000 7fdcc0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.395+0000 7fdcc0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:47.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.395+0000 7fdcc0ff9640 10 monclient: dump: 2024-03-25T18:19:47.397 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:47.397 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:47.397 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:47.397 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:47.397 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.397 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:47.397 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:47.398 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:47.398 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:47.398 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:47.398 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.395+0000 7fdcc27fc640 4 set_mon_vals no callback set 2024-03-25T18:19:47.398 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.396+0000 7fdcc0ff9640 1 -- 172.21.15.122:0/3394619778 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fdcbc004000 con 0x7fdcc414be90 2024-03-25T18:19:47.398 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.397+0000 7fdcc3fff640 10 monclient: get_auth_request con 0x7fdcb006dd90 auth_method 0 2024-03-25T18:19:47.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.397+0000 7fdcc0ff9640 10 client.4380.objecter ms_handle_connect 0x7fdcb006dd90 2024-03-25T18:19:47.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.402+0000 7fdcc0ff9640 1 -- 172.21.15.122:0/3394619778 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fdcbc054aa0 con 0x7fdcc414be90 2024-03-25T18:19:47.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.402+0000 7fdcc0ff9640 10 client.4380.objecter ms_dispatch 0x7fdcc4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:47.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.402+0000 7fdcc0ff9640 3 client.4380.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:47.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.402+0000 7fdcc0ff9640 3 client.4380.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:47.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.402+0000 7fdcc0ff9640 20 client.4380.objecter dump_active .. 0 homeless 2024-03-25T18:19:47.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.402+0000 7fdcc0ff9640 1 -- 172.21.15.122:0/3394619778 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdcbc0505b0 con 0x7fdcc414be90 2024-03-25T18:19:47.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.402+0000 7fdcc0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:47.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.402+0000 7fdcc0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:47.662 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.660+0000 7fdcca7ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:47.662 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.660+0000 7fdcca7ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:47.662 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.660+0000 7fdcca7ed640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:47.662 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.660+0000 7fdcca7ed640 1 -- 172.21.15.122:0/3394619778 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdcc40627b0 con 0x7fdcc414be90 2024-03-25T18:19:47.687 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.685+0000 7fdcc0ff9640 1 -- 172.21.15.122:0/3394619778 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fdcbc0524e0 con 0x7fdcc414be90 2024-03-25T18:19:47.687 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.686+0000 7fdcc0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:47.687 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.686+0000 7fdcc0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:47.687 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:19:47.687 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:47.690 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.688+0000 7fdcca7ed640 10 librados: watch_flush enter 2024-03-25T18:19:47.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.689+0000 7fdcca7ed640 10 librados: watch_flush exit 2024-03-25T18:19:47.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.689+0000 7fdcca7ed640 20 client.4380.objecter shutdown clearing up homeless session... 2024-03-25T18:19:47.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.689+0000 7fdcca7ed640 10 client.4380.objecter successfully canceled tick 2024-03-25T18:19:47.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.689+0000 7fdcca7ed640 1 -- 172.21.15.122:0/3394619778 >> v1:172.21.15.204:6800/676228005 conn(0x7fdcb006dd90 legacy=0x7fdcb0070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.689+0000 7fdcca7ed640 10 monclient: shutdown 2024-03-25T18:19:47.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.689+0000 7fdcca7ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:47.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.689+0000 7fdcca7ed640 1 -- 172.21.15.122:0/3394619778 >> v1:172.21.15.122:6789/0 conn(0x7fdcc414be90 legacy=0x7fdcc4142c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.689+0000 7fdcca7ed640 1 -- 172.21.15.122:0/3394619778 shutdown_connections 2024-03-25T18:19:47.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.689+0000 7fdcca7ed640 1 -- 172.21.15.122:0/3394619778 >> 172.21.15.122:0/3394619778 conn(0x7fdcc40617a0 msgr2=0x7fdcc413f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:47.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.689+0000 7fdcca7ed640 1 -- 172.21.15.122:0/3394619778 shutdown_connections 2024-03-25T18:19:47.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.689+0000 7fdcca7ed640 1 -- 172.21.15.122:0/3394619778 wait complete. 2024-03-25T18:19:47.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:47.689+0000 7fdcca7ed640 1 librados: shutdown 2024-03-25T18:19:47.702 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:54.508 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:54.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.618+0000 7fa55bbf6640 10 monclient: get_monmap_and_config 2024-03-25T18:19:54.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.618+0000 7fa55bbf6640 10 monclient: build_initial_monmap 2024-03-25T18:19:54.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.618+0000 7fa55bbf6640 10 monclient: monmap: 2024-03-25T18:19:54.619 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:19:54.620 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:54.620 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:19:54.619365+0000 2024-03-25T18:19:54.620 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:19:54.619365+0000 2024-03-25T18:19:54.620 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:54.620 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:54.620 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:19:54.620 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:19:54.620 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:19:54.620 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:54.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.618+0000 7fa55bbf6640 1 Processor -- start 2024-03-25T18:19:54.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.619+0000 7fa55bbf6640 1 -- start start 2024-03-25T18:19:54.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.619+0000 7fa55bbf6640 10 monclient: init 2024-03-25T18:19:54.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.620+0000 7fa55bbf6640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:54.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.620+0000 7fa55bbf6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:54.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.620+0000 7fa55bbf6640 10 monclient(hunting): picked mon.noname-b con 0x7fa5540a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:19:54.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.620+0000 7fa55bbf6640 10 monclient(hunting): picked mon.noname-a con 0x7fa55413f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:19:54.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.620+0000 7fa55bbf6640 10 monclient(hunting): picked mon.noname-c con 0x7fa5541430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:19:54.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.620+0000 7fa55bbf6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa554142520 con 0x7fa55413f710 2024-03-25T18:19:54.623 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.620+0000 7fa55bbf6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5540b28a0 con 0x7fa5541430c0 2024-03-25T18:19:54.623 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.621+0000 7fa55bbf6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5540b2a30 con 0x7fa5540a98f0 2024-03-25T18:19:54.623 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.621+0000 7fa55bbf6640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:54.623 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.621+0000 7fa55bbf6640 10 monclient(hunting): authenticate will time out at 7517.887167s 2024-03-25T18:19:54.632 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.631+0000 7fa559992640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7fa5540a98f0 0x7fa5540a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:41188/0 (socket says 172.21.15.122:41188) 2024-03-25T18:19:54.632 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.631+0000 7fa559992640 1 -- 172.21.15.122:0/3233728545 learned_addr learned my addr 172.21.15.122:0/3233728545 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:19:54.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.642+0000 7fa558990640 1 -- 172.21.15.122:0/3233728545 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 199407427 0 0) 0x7fa5540b2a30 con 0x7fa5540a98f0 2024-03-25T18:19:54.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.642+0000 7fa558990640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.642+0000 7fa558990640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.642+0000 7fa558990640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.642+0000 7fa558990640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:19:54.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.642+0000 7fa558990640 1 -- 172.21.15.122:0/3233728545 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa540003520 con 0x7fa5540a98f0 2024-03-25T18:19:54.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.646+0000 7fa558990640 1 -- 172.21.15.122:0/3233728545 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4021258336 0 0) 0x7fa540003520 con 0x7fa5540a98f0 2024-03-25T18:19:54.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.646+0000 7fa558990640 1 -- 172.21.15.122:0/3233728545 >> v1:172.21.15.204:6789/0 conn(0x7fa5541430c0 legacy=0x7fa55414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.646+0000 7fa558990640 1 -- 172.21.15.122:0/3233728545 >> v1:172.21.15.122:6789/0 conn(0x7fa55413f710 legacy=0x7fa5541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.646+0000 7fa558990640 10 monclient: _finish_hunting 0 2024-03-25T18:19:54.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.646+0000 7fa558990640 1 monclient: found mon.noname-b 2024-03-25T18:19:54.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.646+0000 7fa558990640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:54.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.646+0000 7fa558990640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:19:54.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.646+0000 7fa558990640 1 -- 172.21.15.122:0/3233728545 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5540b44c0 con 0x7fa5540a98f0 2024-03-25T18:19:54.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.646+0000 7fa558990640 10 monclient: _finish_auth 0 2024-03-25T18:19:54.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.646+0000 7fa558990640 10 monclient: _check_auth_tickets 2024-03-25T18:19:54.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.646+0000 7fa558990640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:54.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.646+0000 7fa558990640 1 -- 172.21.15.122:0/3233728545 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fa550001690 con 0x7fa5540a98f0 2024-03-25T18:19:54.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.646+0000 7fa558990640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.646+0000 7fa558990640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:54.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.646+0000 7fa558990640 10 monclient: dump: 2024-03-25T18:19:54.648 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:54.648 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:54.648 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:54.649 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:54.649 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.649 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:54.649 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:54.649 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:54.649 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:54.649 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:54.649 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.646+0000 7fa55bbf6640 5 monclient: authenticate success, global_id 4361 2024-03-25T18:19:54.649 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.646+0000 7fa55bbf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:54.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.681+0000 7fa558990640 1 -- 172.21.15.122:0/3233728545 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5500017f0 con 0x7fa5540a98f0 2024-03-25T18:19:54.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.681+0000 7fa558990640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:54.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.681+0000 7fa558990640 1 -- 172.21.15.122:0/3233728545 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fa550001c40 con 0x7fa5540a98f0 2024-03-25T18:19:54.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.681+0000 7fa55bbf6640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:54.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.681+0000 7fa55bbf6640 4 set_mon_vals no callback set 2024-03-25T18:19:54.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.681+0000 7fa558990640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.681+0000 7fa558990640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:54.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.681+0000 7fa558990640 10 monclient: dump: 2024-03-25T18:19:54.682 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:54.682 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:54.683 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:54.683 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:54.683 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.683 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:54.683 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:54.683 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:54.683 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:54.683 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.681+0000 7fa55bbf6640 10 monclient: shutdown 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.681+0000 7fa55bbf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.681+0000 7fa55bbf6640 1 -- 172.21.15.122:0/3233728545 >> v1:172.21.15.122:6790/0 conn(0x7fa5540a98f0 legacy=0x7fa5540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.681+0000 7fa55bbf6640 1 -- 172.21.15.122:0/3233728545 shutdown_connections 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.681+0000 7fa55bbf6640 1 -- 172.21.15.122:0/3233728545 >> 172.21.15.122:0/3233728545 conn(0x7fa5540b7ea0 msgr2=0x7fa5540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.682+0000 7fa55bbf6640 1 -- 172.21.15.122:0/3233728545 shutdown_connections 2024-03-25T18:19:54.685 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.682+0000 7fa55bbf6640 1 -- 172.21.15.122:0/3233728545 wait complete. 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.682+0000 7fa55bbf6640 10 monclient: build_initial_monmap 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.682+0000 7fa55bbf6640 10 monclient: monmap: 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.683+0000 7fa55bbf6640 1 librados: starting msgr at 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.683+0000 7fa55bbf6640 1 librados: starting objecter 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.683+0000 7fa55bbf6640 1 Processor -- start 2024-03-25T18:19:54.686 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.683+0000 7fa55bbf6640 1 -- start start 2024-03-25T18:19:54.687 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.683+0000 7fa55bbf6640 1 librados: setting wanted keys 2024-03-25T18:19:54.687 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.683+0000 7fa55bbf6640 1 librados: calling monclient init 2024-03-25T18:19:54.687 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.683+0000 7fa55bbf6640 10 monclient: init 2024-03-25T18:19:54.687 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.684+0000 7fa55bbf6640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:54.687 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.684+0000 7fa55bbf6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:54.687 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.684+0000 7fa55bbf6640 10 monclient(hunting): picked mon.noname-c con 0x7fa5540a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:19:54.687 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.684+0000 7fa55bbf6640 10 monclient(hunting): picked mon.noname-a con 0x7fa55413f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:19:54.687 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.684+0000 7fa55bbf6640 10 monclient(hunting): picked mon.noname-b con 0x7fa5541430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:19:54.687 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.684+0000 7fa55bbf6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa554138b40 con 0x7fa55413f710 2024-03-25T18:19:54.687 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.684+0000 7fa55bbf6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa554138e20 con 0x7fa5541430c0 2024-03-25T18:19:54.687 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.684+0000 7fa55bbf6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa554139100 con 0x7fa5540a98f0 2024-03-25T18:19:54.687 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.684+0000 7fa55bbf6640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:54.687 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.684+0000 7fa55bbf6640 10 monclient(hunting): authenticate will time out at 7517.950239s 2024-03-25T18:19:54.688 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.685+0000 7fa55a193640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fa5541430c0 0x7fa5541e1fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:34382/0 (socket says 172.21.15.122:34382) 2024-03-25T18:19:54.688 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.685+0000 7fa55a193640 1 -- 172.21.15.122:0/2295095529 learned_addr learned my addr 172.21.15.122:0/2295095529 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:19:54.688 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.685+0000 7fa54a7fc640 10 client.?.objecter ms_handle_connect 0x7fa5541430c0 2024-03-25T18:19:54.688 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.685+0000 7fa54a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:54.688 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.687+0000 7fa54a7fc640 10 client.?.objecter ms_handle_connect 0x7fa5540a98f0 2024-03-25T18:19:54.688 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.687+0000 7fa54a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:54.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.738+0000 7fa54a7fc640 10 client.?.objecter ms_handle_connect 0x7fa55413f710 2024-03-25T18:19:54.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.738+0000 7fa54a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:54.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.750+0000 7fa54a7fc640 1 -- 172.21.15.122:0/2295095529 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4266960059 0 0) 0x7fa554138b40 con 0x7fa55413f710 2024-03-25T18:19:54.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.750+0000 7fa54a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.750+0000 7fa54a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.750+0000 7fa54a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.750+0000 7fa54a7fc640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:19:54.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.750+0000 7fa54a7fc640 1 -- 172.21.15.122:0/2295095529 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa538004eb0 con 0x7fa55413f710 2024-03-25T18:19:54.768 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.767+0000 7fa54a7fc640 1 -- 172.21.15.122:0/2295095529 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1659658594 0 0) 0x7fa538004eb0 con 0x7fa55413f710 2024-03-25T18:19:54.768 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.767+0000 7fa54a7fc640 1 -- 172.21.15.122:0/2295095529 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa554138b40 con 0x7fa55413f710 2024-03-25T18:19:54.768 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.767+0000 7fa54a7fc640 1 -- 172.21.15.122:0/2295095529 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fa5440018c0 con 0x7fa55413f710 2024-03-25T18:19:54.768 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.767+0000 7fa54a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.769 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.767+0000 7fa54a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:54.769 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.767+0000 7fa54a7fc640 10 monclient(hunting): dump: 2024-03-25T18:19:54.769 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:54.769 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:54.769 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:54.769 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:54.769 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.769 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:54.769 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:54.769 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:54.769 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:54.769 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:54.769 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.767+0000 7fa54a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:54.778 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.776+0000 7fa54a7fc640 1 -- 172.21.15.122:0/2295095529 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2803043328 0 0) 0x7fa554138b40 con 0x7fa55413f710 2024-03-25T18:19:54.778 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.776+0000 7fa54a7fc640 1 -- 172.21.15.122:0/2295095529 >> v1:172.21.15.122:6790/0 conn(0x7fa5540a98f0 legacy=0x7fa554137380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.778 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.776+0000 7fa54a7fc640 1 -- 172.21.15.122:0/2295095529 >> v1:172.21.15.204:6789/0 conn(0x7fa5541430c0 legacy=0x7fa5541e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.778 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.776+0000 7fa54a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:54.778 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.776+0000 7fa54a7fc640 1 monclient: found mon.a 2024-03-25T18:19:54.778 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.777+0000 7fa54a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:54.778 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.777+0000 7fa54a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:54.778 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.777+0000 7fa54a7fc640 1 -- 172.21.15.122:0/2295095529 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5541397a0 con 0x7fa55413f710 2024-03-25T18:19:54.778 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.777+0000 7fa54a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:54.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.777+0000 7fa54a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:54.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.777+0000 7fa54a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:54.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.777+0000 7fa55bbf6640 5 monclient: authenticate success, global_id 4386 2024-03-25T18:19:54.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.777+0000 7fa55bbf6640 10 monclient: _renew_subs 2024-03-25T18:19:54.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.777+0000 7fa55bbf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:54.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.777+0000 7fa55bbf6640 1 -- 172.21.15.122:0/2295095529 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa554136aa0 con 0x7fa55413f710 2024-03-25T18:19:54.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.777+0000 7fa55bbf6640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:54.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.777+0000 7fa55bbf6640 10 monclient: _renew_subs 2024-03-25T18:19:54.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.777+0000 7fa55bbf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:54.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.777+0000 7fa55bbf6640 1 -- 172.21.15.122:0/2295095529 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa55413c320 con 0x7fa55413f710 2024-03-25T18:19:54.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.777+0000 7fa55bbf6640 1 librados: init done 2024-03-25T18:19:54.780 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.778+0000 7fa55bbf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:54.780 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.778+0000 7fa55bbf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:54.780 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.778+0000 7fa55bbf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:54.780 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:54.778+0000 7fa55bbf6640 1 -- 172.21.15.122:0/2295095529 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5541395b0 con 0x7fa55413f710 2024-03-25T18:19:55.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.036+0000 7fa54a7fc640 1 -- 172.21.15.122:0/2295095529 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5440035f0 con 0x7fa55413f710 2024-03-25T18:19:55.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.036+0000 7fa54a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:55.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.036+0000 7fa54a7fc640 1 -- 172.21.15.122:0/2295095529 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fa5440039c0 con 0x7fa55413f710 2024-03-25T18:19:55.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.036+0000 7fa54a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:55.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.036+0000 7fa54a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:55.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.036+0000 7fa54a7fc640 10 monclient: dump: 2024-03-25T18:19:55.038 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:19:55.038 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:19:55.038 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:55.039 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:19:55.039 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:55.039 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:19:55.039 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:19:55.039 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:19:55.039 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:19:55.039 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:19:55.039 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.036+0000 7fa54bfff640 4 set_mon_vals no callback set 2024-03-25T18:19:55.040 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.037+0000 7fa54a7fc640 1 -- 172.21.15.122:0/2295095529 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fa544003be0 con 0x7fa55413f710 2024-03-25T18:19:55.040 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.038+0000 7fa559992640 10 monclient: get_auth_request con 0x7fa53806dd70 auth_method 0 2024-03-25T18:19:55.040 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.038+0000 7fa54a7fc640 10 client.4386.objecter ms_handle_connect 0x7fa53806dd70 2024-03-25T18:19:55.045 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.044+0000 7fa54a7fc640 1 -- 172.21.15.122:0/2295095529 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fa544053570 con 0x7fa55413f710 2024-03-25T18:19:55.045 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.044+0000 7fa54a7fc640 10 client.4386.objecter ms_dispatch 0x7fa554000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:55.045 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.044+0000 7fa54a7fc640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:55.045 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.044+0000 7fa54a7fc640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:55.045 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.044+0000 7fa54a7fc640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-25T18:19:55.045 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.044+0000 7fa54a7fc640 1 -- 172.21.15.122:0/2295095529 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa54404f080 con 0x7fa55413f710 2024-03-25T18:19:55.045 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.044+0000 7fa54a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:55.046 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.044+0000 7fa54a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:55.274 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.273+0000 7fa55bbf6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:55.274 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.273+0000 7fa55bbf6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:55.274 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.273+0000 7fa55bbf6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:19:55.274 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.273+0000 7fa55bbf6640 1 -- 172.21.15.122:0/2295095529 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5540627b0 con 0x7fa55413f710 2024-03-25T18:19:55.316 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.315+0000 7fa54a7fc640 1 -- 172.21.15.122:0/2295095529 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fa544050fb0 con 0x7fa55413f710 2024-03-25T18:19:55.316 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.315+0000 7fa54a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:55.316 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.315+0000 7fa54a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:55.316 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:19:55.317 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:55.319 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.317+0000 7fa55bbf6640 10 librados: watch_flush enter 2024-03-25T18:19:55.319 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.318+0000 7fa55bbf6640 10 librados: watch_flush exit 2024-03-25T18:19:55.319 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.318+0000 7fa55bbf6640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-25T18:19:55.319 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.318+0000 7fa55bbf6640 10 client.4386.objecter successfully canceled tick 2024-03-25T18:19:55.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.318+0000 7fa55bbf6640 1 -- 172.21.15.122:0/2295095529 >> v1:172.21.15.204:6800/676228005 conn(0x7fa53806dd70 legacy=0x7fa538070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:55.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.318+0000 7fa55bbf6640 10 monclient: shutdown 2024-03-25T18:19:55.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.318+0000 7fa55bbf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:55.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.318+0000 7fa55bbf6640 1 -- 172.21.15.122:0/2295095529 >> v1:172.21.15.122:6789/0 conn(0x7fa55413f710 legacy=0x7fa554137a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:55.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.318+0000 7fa55bbf6640 1 -- 172.21.15.122:0/2295095529 shutdown_connections 2024-03-25T18:19:55.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.318+0000 7fa55bbf6640 1 -- 172.21.15.122:0/2295095529 >> 172.21.15.122:0/2295095529 conn(0x7fa5540b7ea0 msgr2=0x7fa55414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:55.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.318+0000 7fa55bbf6640 1 -- 172.21.15.122:0/2295095529 shutdown_connections 2024-03-25T18:19:55.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.318+0000 7fa55bbf6640 1 -- 172.21.15.122:0/2295095529 wait complete. 2024-03-25T18:19:55.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:19:55.318+0000 7fa55bbf6640 1 librados: shutdown 2024-03-25T18:19:55.331 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:02.137 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:02.248 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.246+0000 7ff5a3af6640 10 monclient: get_monmap_and_config 2024-03-25T18:20:02.248 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.246+0000 7ff5a3af6640 10 monclient: build_initial_monmap 2024-03-25T18:20:02.248 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.246+0000 7ff5a3af6640 10 monclient: monmap: 2024-03-25T18:20:02.248 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:20:02.248 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:02.248 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:20:02.247954+0000 2024-03-25T18:20:02.248 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:20:02.247954+0000 2024-03-25T18:20:02.248 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:02.248 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:02.248 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:20:02.248 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:20:02.248 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:20:02.248 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.247+0000 7ff5a3af6640 1 Processor -- start 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.247+0000 7ff5a3af6640 1 -- start start 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.247+0000 7ff5a3af6640 10 monclient: init 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.248+0000 7ff5a3af6640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.248+0000 7ff5a3af6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.248+0000 7ff5a3af6640 10 monclient(hunting): picked mon.noname-a con 0x7ff59c070410 addr v1:172.21.15.122:6789/0 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.248+0000 7ff5a3af6640 10 monclient(hunting): picked mon.noname-b con 0x7ff59c0707f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.248+0000 7ff5a3af6640 10 monclient(hunting): picked mon.noname-c con 0x7ff59c13db40 addr v1:172.21.15.204:6789/0 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.248+0000 7ff5a3af6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c0adea0 con 0x7ff59c070410 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.248+0000 7ff5a3af6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c1445a0 con 0x7ff59c13db40 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.248+0000 7ff5a3af6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c0720a0 con 0x7ff59c0707f0 2024-03-25T18:20:02.250 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.248+0000 7ff5a3af6640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:02.251 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.248+0000 7ff5a3af6640 10 monclient(hunting): authenticate will time out at 7525.514681s 2024-03-25T18:20:02.259 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.258+0000 7ff5a1892640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff59c070410 0x7ff59c143e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:57226/0 (socket says 172.21.15.122:57226) 2024-03-25T18:20:02.259 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.258+0000 7ff5a1892640 1 -- 172.21.15.122:0/3055522042 learned_addr learned my addr 172.21.15.122:0/3055522042 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:20:02.276 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.275+0000 7ff5a0890640 1 -- 172.21.15.122:0/3055522042 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320260051 0 0) 0x7ff59c0adea0 con 0x7ff59c070410 2024-03-25T18:20:02.276 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.275+0000 7ff5a0890640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.276 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.275+0000 7ff5a0890640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.276 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.275+0000 7ff5a0890640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.276 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.275+0000 7ff5a0890640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:20:02.276 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.275+0000 7ff5a0890640 1 -- 172.21.15.122:0/3055522042 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff588003520 con 0x7ff59c070410 2024-03-25T18:20:02.287 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.286+0000 7ff5a0890640 1 -- 172.21.15.122:0/3055522042 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3782202470 0 0) 0x7ff59c1445a0 con 0x7ff59c13db40 2024-03-25T18:20:02.287 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.286+0000 7ff5a0890640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.287 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.286+0000 7ff5a0890640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.287 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.286+0000 7ff5a0890640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.287 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.286+0000 7ff5a0890640 10 monclient(hunting): my global_id is 4387 2024-03-25T18:20:02.287 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.286+0000 7ff5a0890640 1 -- 172.21.15.122:0/3055522042 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff59c0adea0 con 0x7ff59c13db40 2024-03-25T18:20:02.287 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.286+0000 7ff5a0890640 1 -- 172.21.15.122:0/3055522042 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530380477 0 0) 0x7ff59c0720a0 con 0x7ff59c0707f0 2024-03-25T18:20:02.288 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.286+0000 7ff5a0890640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.288 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.286+0000 7ff5a0890640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.288 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.286+0000 7ff5a0890640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.288 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.286+0000 7ff5a0890640 10 monclient(hunting): my global_id is 4367 2024-03-25T18:20:02.288 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.286+0000 7ff5a0890640 1 -- 172.21.15.122:0/3055522042 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff59c1445a0 con 0x7ff59c0707f0 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a0890640 1 -- 172.21.15.122:0/3055522042 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 120034535 0 0) 0x7ff59c1445a0 con 0x7ff59c0707f0 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a0890640 1 -- 172.21.15.122:0/3055522042 >> v1:172.21.15.204:6789/0 conn(0x7ff59c13db40 legacy=0x7ff59c14ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a0890640 1 -- 172.21.15.122:0/3055522042 >> v1:172.21.15.122:6789/0 conn(0x7ff59c070410 legacy=0x7ff59c143e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a0890640 10 monclient: _finish_hunting 0 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a0890640 1 monclient: found mon.noname-b 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a0890640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a0890640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a0890640 1 -- 172.21.15.122:0/3055522042 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c072380 con 0x7ff59c0707f0 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a0890640 10 monclient: _finish_auth 0 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a0890640 10 monclient: _check_auth_tickets 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a0890640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:02.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a0890640 1 -- 172.21.15.122:0/3055522042 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff58c0015b0 con 0x7ff59c0707f0 2024-03-25T18:20:02.301 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a3af6640 5 monclient: authenticate success, global_id 4367 2024-03-25T18:20:02.301 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a3af6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:02.301 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a0890640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.301 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a0890640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:02.301 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a0890640 10 monclient: dump: 2024-03-25T18:20:02.301 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:02.301 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:02.301 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:02.301 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:02.301 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.301 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:02.301 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:02.301 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:02.301 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:02.301 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:02.301 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.298+0000 7ff5a3af6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.322+0000 7ff5a0890640 1 -- 172.21.15.122:0/3055522042 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff58c001710 con 0x7ff59c0707f0 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.322+0000 7ff5a0890640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.322+0000 7ff5a0890640 1 -- 172.21.15.122:0/3055522042 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff58c001b60 con 0x7ff59c0707f0 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.322+0000 7ff5a3af6640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.322+0000 7ff5a3af6640 4 set_mon_vals no callback set 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.322+0000 7ff5a0890640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.322+0000 7ff5a0890640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.322+0000 7ff5a0890640 10 monclient: dump: 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:02.323 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:02.324 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.324 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:02.324 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:02.324 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:02.324 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:02.324 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:02.326 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.322+0000 7ff5a3af6640 10 monclient: shutdown 2024-03-25T18:20:02.326 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.322+0000 7ff5a3af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:02.326 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.322+0000 7ff5a3af6640 1 -- 172.21.15.122:0/3055522042 >> v1:172.21.15.122:6790/0 conn(0x7ff59c0707f0 legacy=0x7ff59c13d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.326 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.323+0000 7ff5a3af6640 1 -- 172.21.15.122:0/3055522042 shutdown_connections 2024-03-25T18:20:02.326 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.323+0000 7ff5a3af6640 1 -- 172.21.15.122:0/3055522042 >> 172.21.15.122:0/3055522042 conn(0x7ff59c06b630 msgr2=0x7ff59c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.323+0000 7ff5a3af6640 1 -- 172.21.15.122:0/3055522042 shutdown_connections 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.323+0000 7ff5a3af6640 1 -- 172.21.15.122:0/3055522042 wait complete. 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.323+0000 7ff5a3af6640 10 monclient: build_initial_monmap 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.323+0000 7ff5a3af6640 10 monclient: monmap: 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:20:02.327 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:02.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.324+0000 7ff5a3af6640 1 librados: starting msgr at 2024-03-25T18:20:02.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.324+0000 7ff5a3af6640 1 librados: starting objecter 2024-03-25T18:20:02.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.324+0000 7ff5a3af6640 1 Processor -- start 2024-03-25T18:20:02.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.324+0000 7ff5a3af6640 1 -- start start 2024-03-25T18:20:02.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.324+0000 7ff5a3af6640 1 librados: setting wanted keys 2024-03-25T18:20:02.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.324+0000 7ff5a3af6640 1 librados: calling monclient init 2024-03-25T18:20:02.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.324+0000 7ff5a3af6640 10 monclient: init 2024-03-25T18:20:02.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.325+0000 7ff5a3af6640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:02.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.325+0000 7ff5a3af6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:02.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.325+0000 7ff5a3af6640 10 monclient(hunting): picked mon.noname-b con 0x7ff59c070410 addr v1:172.21.15.204:6789/0 2024-03-25T18:20:02.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.325+0000 7ff5a3af6640 10 monclient(hunting): picked mon.noname-c con 0x7ff59c0707f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:20:02.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.325+0000 7ff5a3af6640 10 monclient(hunting): picked mon.noname-a con 0x7ff59c13db40 addr v1:172.21.15.122:6789/0 2024-03-25T18:20:02.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.325+0000 7ff5a3af6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c0ab600 con 0x7ff59c13db40 2024-03-25T18:20:02.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.325+0000 7ff5a3af6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c1d5a20 con 0x7ff59c070410 2024-03-25T18:20:02.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.325+0000 7ff5a3af6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff59c0a9fd0 con 0x7ff59c0707f0 2024-03-25T18:20:02.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.325+0000 7ff5a3af6640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:02.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.325+0000 7ff5a3af6640 10 monclient(hunting): authenticate will time out at 7525.591222s 2024-03-25T18:20:02.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.339+0000 7ff5a1091640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7ff59c0707f0 0x7ff59c1d5f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:39236/0 (socket says 172.21.15.122:39236) 2024-03-25T18:20:02.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.339+0000 7ff5a1091640 1 -- 172.21.15.122:0/1616238895 learned_addr learned my addr 172.21.15.122:0/1616238895 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:20:02.342 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.341+0000 7ff5927fc640 10 client.?.objecter ms_handle_connect 0x7ff59c0707f0 2024-03-25T18:20:02.342 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.341+0000 7ff5927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:02.343 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.342+0000 7ff5927fc640 10 client.?.objecter ms_handle_connect 0x7ff59c070410 2024-03-25T18:20:02.343 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.342+0000 7ff5927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:02.346 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.345+0000 7ff5927fc640 10 client.?.objecter ms_handle_connect 0x7ff59c13db40 2024-03-25T18:20:02.346 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.345+0000 7ff5927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:02.351 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.350+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493770369 0 0) 0x7ff59c0a9fd0 con 0x7ff59c0707f0 2024-03-25T18:20:02.351 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.350+0000 7ff5927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.351 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.350+0000 7ff5927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.351 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.350+0000 7ff5927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.351 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.350+0000 7ff5927fc640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:20:02.351 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.350+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff580004ee0 con 0x7ff59c0707f0 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.352+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003496316 0 0) 0x7ff59c1d5a20 con 0x7ff59c070410 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.352+0000 7ff5927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.352+0000 7ff5927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.352+0000 7ff5927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.352+0000 7ff5927fc640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:20:02.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.352+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff59c0a9fd0 con 0x7ff59c070410 2024-03-25T18:20:02.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.360+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1865606241 0 0) 0x7ff580004ee0 con 0x7ff59c0707f0 2024-03-25T18:20:02.362 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.360+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff59c1d5a20 con 0x7ff59c0707f0 2024-03-25T18:20:02.362 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.360+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff58c001ae0 con 0x7ff59c0707f0 2024-03-25T18:20:02.362 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.360+0000 7ff5927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.362 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.360+0000 7ff5927fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:02.362 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.360+0000 7ff5927fc640 10 monclient(hunting): dump: 2024-03-25T18:20:02.362 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:02.362 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:02.362 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:02.362 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:02.362 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.362 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:02.362 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:02.362 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:02.362 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:02.362 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:02.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.360+0000 7ff5927fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:02.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.361+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739412383 0 0) 0x7ff59c0ab600 con 0x7ff59c13db40 2024-03-25T18:20:02.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.361+0000 7ff5927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.361+0000 7ff5927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.361+0000 7ff5927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.361+0000 7ff5927fc640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:20:02.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.361+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff580004ee0 con 0x7ff59c13db40 2024-03-25T18:20:02.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 717542906 0 0) 0x7ff59c0a9fd0 con 0x7ff59c070410 2024-03-25T18:20:02.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff59c0ab600 con 0x7ff59c070410 2024-03-25T18:20:02.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff598001910 con 0x7ff59c070410 2024-03-25T18:20:02.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5927fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:02.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5927fc640 10 monclient(hunting): dump: 2024-03-25T18:20:02.371 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:02.371 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:02.371 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:02.371 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:02.371 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.371 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:02.371 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:02.371 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:02.371 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:02.371 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:02.371 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1924562790 0 0) 0x7ff59c1d5a20 con 0x7ff59c0707f0 2024-03-25T18:20:02.371 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 >> v1:172.21.15.204:6789/0 conn(0x7ff59c070410 legacy=0x7ff59c0a98c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.371 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 >> v1:172.21.15.122:6789/0 conn(0x7ff59c13db40 legacy=0x7ff59c1e0430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.371 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5927fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:02.371 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5927fc640 1 monclient: found mon.c 2024-03-25T18:20:02.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:02.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5927fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:20:02.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff59c0aa160 con 0x7ff59c0707f0 2024-03-25T18:20:02.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5927fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:02.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5927fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:02.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:02.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5a3af6640 5 monclient: authenticate success, global_id 4370 2024-03-25T18:20:02.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5a3af6640 10 monclient: _renew_subs 2024-03-25T18:20:02.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5a3af6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:20:02.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5a3af6640 1 -- 172.21.15.122:0/1616238895 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff59c1d6650 con 0x7ff59c0707f0 2024-03-25T18:20:02.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5a3af6640 10 client.4370.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:02.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5a3af6640 10 monclient: _renew_subs 2024-03-25T18:20:02.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5a3af6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:20:02.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5a3af6640 1 -- 172.21.15.122:0/1616238895 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff59c1411e0 con 0x7ff59c0707f0 2024-03-25T18:20:02.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.368+0000 7ff5a3af6640 1 librados: init done 2024-03-25T18:20:02.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.369+0000 7ff5a3af6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:02.373 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.369+0000 7ff5a3af6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:02.373 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.369+0000 7ff5a3af6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:20:02.373 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.369+0000 7ff5a3af6640 1 -- 172.21.15.122:0/1616238895 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff59c074d50 con 0x7ff59c0707f0 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.671+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff58c0038a0 con 0x7ff59c0707f0 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.671+0000 7ff5927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.671+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff58c003cb0 con 0x7ff59c0707f0 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.671+0000 7ff5927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.671+0000 7ff5927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.671+0000 7ff5927fc640 10 monclient: dump: 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:02.673 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:02.673 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.673 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:02.673 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:02.673 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:02.673 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:02.673 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:02.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.671+0000 7ff593fff640 4 set_mon_vals no callback set 2024-03-25T18:20:02.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.671+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7ff58c003ed0 con 0x7ff59c0707f0 2024-03-25T18:20:02.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.672+0000 7ff5a1892640 10 monclient: get_auth_request con 0x7ff58006e020 auth_method 0 2024-03-25T18:20:02.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.672+0000 7ff5927fc640 10 client.4370.objecter ms_handle_connect 0x7ff58006e020 2024-03-25T18:20:02.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.678+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7ff58c001190 con 0x7ff59c0707f0 2024-03-25T18:20:02.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.678+0000 7ff5927fc640 10 client.4370.objecter ms_dispatch 0x7ff59c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:02.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.678+0000 7ff5927fc640 3 client.4370.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:02.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.678+0000 7ff5927fc640 3 client.4370.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:02.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.679+0000 7ff5927fc640 20 client.4370.objecter dump_active .. 0 homeless 2024-03-25T18:20:02.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.679+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff58c0503e0 con 0x7ff59c0707f0 2024-03-25T18:20:02.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.679+0000 7ff5927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:02.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.679+0000 7ff5927fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:02.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.932+0000 7ff5a3af6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:02.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.932+0000 7ff5a3af6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:02.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.932+0000 7ff5a3af6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:20:02.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:02.932+0000 7ff5a3af6640 1 -- 172.21.15.122:0/1616238895 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff59c0627b0 con 0x7ff59c0707f0 2024-03-25T18:20:03.106 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:03.105+0000 7ff5927fc640 1 -- 172.21.15.122:0/1616238895 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7ff58c052310 con 0x7ff59c0707f0 2024-03-25T18:20:03.106 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:03.105+0000 7ff5927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:03.106 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:03.105+0000 7ff5927fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:03.106 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:20:03.106 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:03.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:03.107+0000 7ff5a3af6640 10 librados: watch_flush enter 2024-03-25T18:20:03.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:03.107+0000 7ff5a3af6640 10 librados: watch_flush exit 2024-03-25T18:20:03.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:03.107+0000 7ff5a3af6640 20 client.4370.objecter shutdown clearing up homeless session... 2024-03-25T18:20:03.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:03.107+0000 7ff5a3af6640 10 client.4370.objecter successfully canceled tick 2024-03-25T18:20:03.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:03.107+0000 7ff5a3af6640 1 -- 172.21.15.122:0/1616238895 >> v1:172.21.15.204:6800/676228005 conn(0x7ff58006e020 legacy=0x7ff5800704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:03.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:03.107+0000 7ff5a3af6640 10 monclient: shutdown 2024-03-25T18:20:03.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:03.107+0000 7ff5a3af6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:03.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:03.107+0000 7ff5a3af6640 1 -- 172.21.15.122:0/1616238895 >> v1:172.21.15.122:6790/0 conn(0x7ff59c0707f0 legacy=0x7ff59c1d5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:03.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:03.107+0000 7ff5a3af6640 1 -- 172.21.15.122:0/1616238895 shutdown_connections 2024-03-25T18:20:03.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:03.108+0000 7ff5a3af6640 1 -- 172.21.15.122:0/1616238895 >> 172.21.15.122:0/1616238895 conn(0x7ff59c06b630 msgr2=0x7ff59c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:03.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:03.108+0000 7ff5a3af6640 1 -- 172.21.15.122:0/1616238895 shutdown_connections 2024-03-25T18:20:03.110 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:03.108+0000 7ff5a3af6640 1 -- 172.21.15.122:0/1616238895 wait complete. 2024-03-25T18:20:03.110 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:03.108+0000 7ff5a3af6640 1 librados: shutdown 2024-03-25T18:20:03.120 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:06.014 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:03:59.756 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:06.023 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:03:59.766 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:06.060 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:03:59.803 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:06.073 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:03:59.816 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:06.083 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:03:59.826 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:06.088 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:03:59.834 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:06.094 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:03:59.839 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:06.113 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:03:59.856 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:06.125 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:03:59.869 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:09.927 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:10.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.039+0000 7f5d55457640 10 monclient: get_monmap_and_config 2024-03-25T18:20:10.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.039+0000 7f5d55457640 10 monclient: build_initial_monmap 2024-03-25T18:20:10.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.039+0000 7f5d55457640 10 monclient: monmap: 2024-03-25T18:20:10.041 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:20:10.041 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:10.042 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:20:10.040998+0000 2024-03-25T18:20:10.042 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:20:10.040998+0000 2024-03-25T18:20:10.042 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:10.042 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:10.042 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:20:10.042 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:20:10.042 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:20:10.042 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:10.042 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.040+0000 7f5d55457640 1 Processor -- start 2024-03-25T18:20:10.042 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.040+0000 7f5d55457640 1 -- start start 2024-03-25T18:20:10.042 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.040+0000 7f5d55457640 10 monclient: init 2024-03-25T18:20:10.043 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.041+0000 7f5d55457640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:10.043 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.041+0000 7f5d55457640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:10.043 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.041+0000 7f5d55457640 10 monclient(hunting): picked mon.noname-c con 0x7f5d500a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:20:10.043 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.041+0000 7f5d55457640 10 monclient(hunting): picked mon.noname-b con 0x7f5d5013f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:20:10.043 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.041+0000 7f5d55457640 10 monclient(hunting): picked mon.noname-a con 0x7f5d501430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:20:10.043 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.041+0000 7f5d55457640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d50142520 con 0x7f5d501430c0 2024-03-25T18:20:10.043 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.041+0000 7f5d55457640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d500b28a0 con 0x7f5d500a98f0 2024-03-25T18:20:10.044 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.041+0000 7f5d55457640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d500b2a30 con 0x7f5d5013f710 2024-03-25T18:20:10.044 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.041+0000 7f5d55457640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:10.044 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.041+0000 7f5d55457640 10 monclient(hunting): authenticate will time out at 7533.307393s 2024-03-25T18:20:10.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.047+0000 7f5d4e7fc640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f5d5013f710 0x7f5d501429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:59440/0 (socket says 172.21.15.122:59440) 2024-03-25T18:20:10.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.047+0000 7f5d4e7fc640 1 -- 172.21.15.122:0/777918620 learned_addr learned my addr 172.21.15.122:0/777918620 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:20:10.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.071+0000 7f5d4dffb640 1 -- 172.21.15.122:0/777918620 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 658750033 0 0) 0x7f5d500b2a30 con 0x7f5d5013f710 2024-03-25T18:20:10.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.071+0000 7f5d4dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.071+0000 7f5d4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.071+0000 7f5d4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.071+0000 7f5d4dffb640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:20:10.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.071+0000 7f5d4dffb640 1 -- 172.21.15.122:0/777918620 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d38003520 con 0x7f5d5013f710 2024-03-25T18:20:10.081 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.079+0000 7f5d4dffb640 1 -- 172.21.15.122:0/777918620 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304519578 0 0) 0x7f5d500b28a0 con 0x7f5d500a98f0 2024-03-25T18:20:10.081 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.079+0000 7f5d4dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.081 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.079+0000 7f5d4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.081 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.079+0000 7f5d4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.081 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.079+0000 7f5d4dffb640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:20:10.081 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.079+0000 7f5d4dffb640 1 -- 172.21.15.122:0/777918620 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d500b2a30 con 0x7f5d500a98f0 2024-03-25T18:20:10.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.083+0000 7f5d4dffb640 1 -- 172.21.15.122:0/777918620 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 321731987 0 0) 0x7f5d50142520 con 0x7f5d501430c0 2024-03-25T18:20:10.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.083+0000 7f5d4dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.083+0000 7f5d4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.083+0000 7f5d4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.083+0000 7f5d4dffb640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:20:10.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.083+0000 7f5d4dffb640 1 -- 172.21.15.122:0/777918620 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d500b28a0 con 0x7f5d501430c0 2024-03-25T18:20:10.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d4dffb640 1 -- 172.21.15.122:0/777918620 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1538237816 0 0) 0x7f5d500b28a0 con 0x7f5d501430c0 2024-03-25T18:20:10.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d4dffb640 1 -- 172.21.15.122:0/777918620 >> v1:172.21.15.122:6790/0 conn(0x7f5d5013f710 legacy=0x7f5d501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d4dffb640 1 -- 172.21.15.122:0/777918620 >> v1:172.21.15.204:6789/0 conn(0x7f5d500a98f0 legacy=0x7f5d500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d4dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:20:10.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d4dffb640 1 monclient: found mon.noname-a 2024-03-25T18:20:10.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d4dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:10.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d4dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:20:10.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d4dffb640 1 -- 172.21.15.122:0/777918620 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d500b44c0 con 0x7f5d501430c0 2024-03-25T18:20:10.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d4dffb640 10 monclient: _finish_auth 0 2024-03-25T18:20:10.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d4dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:20:10.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:10.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d4dffb640 1 -- 172.21.15.122:0/777918620 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5d48001930 con 0x7f5d501430c0 2024-03-25T18:20:10.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d55457640 5 monclient: authenticate success, global_id 4395 2024-03-25T18:20:10.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d55457640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:10.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:10.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d4dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:10.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d4dffb640 10 monclient: dump: 2024-03-25T18:20:10.103 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:10.103 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:10.103 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:10.103 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:10.103 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:10.104 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:10.104 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:10.104 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:10.104 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:10.104 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:10.104 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.101+0000 7f5d55457640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:10.115 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.114+0000 7f5d4dffb640 1 -- 172.21.15.122:0/777918620 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d48001a90 con 0x7f5d501430c0 2024-03-25T18:20:10.115 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.114+0000 7f5d4dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:10.115 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.114+0000 7f5d4dffb640 1 -- 172.21.15.122:0/777918620 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5d48001ee0 con 0x7f5d501430c0 2024-03-25T18:20:10.115 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.114+0000 7f5d4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:10.116 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.114+0000 7f5d4dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:10.116 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.114+0000 7f5d4dffb640 10 monclient: dump: 2024-03-25T18:20:10.116 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:10.116 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:10.116 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:10.116 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:10.116 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:10.116 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:10.116 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:10.116 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:10.116 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:10.116 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:10.116 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.114+0000 7f5d55457640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:10.116 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.114+0000 7f5d55457640 4 set_mon_vals no callback set 2024-03-25T18:20:10.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.115+0000 7f5d55457640 10 monclient: shutdown 2024-03-25T18:20:10.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.115+0000 7f5d55457640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:10.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.115+0000 7f5d55457640 1 -- 172.21.15.122:0/777918620 >> v1:172.21.15.122:6789/0 conn(0x7f5d501430c0 legacy=0x7f5d5014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.115+0000 7f5d55457640 1 -- 172.21.15.122:0/777918620 shutdown_connections 2024-03-25T18:20:10.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.115+0000 7f5d55457640 1 -- 172.21.15.122:0/777918620 >> 172.21.15.122:0/777918620 conn(0x7f5d500b7ea0 msgr2=0x7f5d500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:10.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.115+0000 7f5d55457640 1 -- 172.21.15.122:0/777918620 shutdown_connections 2024-03-25T18:20:10.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.115+0000 7f5d55457640 1 -- 172.21.15.122:0/777918620 wait complete. 2024-03-25T18:20:10.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.116+0000 7f5d55457640 10 monclient: build_initial_monmap 2024-03-25T18:20:10.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.116+0000 7f5d55457640 10 monclient: monmap: 2024-03-25T18:20:10.118 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:20:10.118 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:10.118 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:20:10.118 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:20:10.118 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:10.118 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:20:10.119 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:20:10.119 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:20:10.119 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:20:10.119 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:10.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.116+0000 7f5d55457640 1 librados: starting msgr at 2024-03-25T18:20:10.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.116+0000 7f5d55457640 1 librados: starting objecter 2024-03-25T18:20:10.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.116+0000 7f5d55457640 1 Processor -- start 2024-03-25T18:20:10.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.116+0000 7f5d55457640 1 -- start start 2024-03-25T18:20:10.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.116+0000 7f5d55457640 1 librados: setting wanted keys 2024-03-25T18:20:10.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.116+0000 7f5d55457640 1 librados: calling monclient init 2024-03-25T18:20:10.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.116+0000 7f5d55457640 10 monclient: init 2024-03-25T18:20:10.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.117+0000 7f5d55457640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:10.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.117+0000 7f5d55457640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:10.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.117+0000 7f5d55457640 10 monclient(hunting): picked mon.noname-a con 0x7f5d500a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:20:10.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.117+0000 7f5d55457640 10 monclient(hunting): picked mon.noname-c con 0x7f5d5013f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:20:10.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.117+0000 7f5d55457640 10 monclient(hunting): picked mon.noname-b con 0x7f5d501430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:20:10.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.117+0000 7f5d55457640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d501397c0 con 0x7f5d500a98f0 2024-03-25T18:20:10.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.117+0000 7f5d55457640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d501d8260 con 0x7f5d501430c0 2024-03-25T18:20:10.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.117+0000 7f5d55457640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d50138ab0 con 0x7f5d5013f710 2024-03-25T18:20:10.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.117+0000 7f5d55457640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:10.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.117+0000 7f5d55457640 10 monclient(hunting): authenticate will time out at 7533.383476s 2024-03-25T18:20:10.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.132+0000 7f5d4f7fe640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f5d501430c0 0x7f5d50134010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:48906/0 (socket says 172.21.15.122:48906) 2024-03-25T18:20:10.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.132+0000 7f5d4f7fe640 1 -- 172.21.15.122:0/3760265981 learned_addr learned my addr 172.21.15.122:0/3760265981 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:20:10.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.132+0000 7f5d377fe640 10 client.?.objecter ms_handle_connect 0x7f5d501430c0 2024-03-25T18:20:10.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.132+0000 7f5d377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:10.137 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.136+0000 7f5d377fe640 10 client.?.objecter ms_handle_connect 0x7f5d500a98f0 2024-03-25T18:20:10.137 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.136+0000 7f5d377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:10.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.138+0000 7f5d377fe640 10 client.?.objecter ms_handle_connect 0x7f5d5013f710 2024-03-25T18:20:10.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.138+0000 7f5d377fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:10.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.149+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2781359391 0 0) 0x7f5d501d8260 con 0x7f5d501430c0 2024-03-25T18:20:10.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.149+0000 7f5d377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.149+0000 7f5d377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.149+0000 7f5d377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.149+0000 7f5d377fe640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:20:10.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.149+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d3c004eb0 con 0x7f5d501430c0 2024-03-25T18:20:10.159 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.158+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2066487331 0 0) 0x7f5d501397c0 con 0x7f5d500a98f0 2024-03-25T18:20:10.159 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.158+0000 7f5d377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.159 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.158+0000 7f5d377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.160 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.158+0000 7f5d377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.160 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.158+0000 7f5d377fe640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:20:10.160 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.158+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d501d8260 con 0x7f5d500a98f0 2024-03-25T18:20:10.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.162+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1293772606 0 0) 0x7f5d50138ab0 con 0x7f5d5013f710 2024-03-25T18:20:10.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.162+0000 7f5d377fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.162+0000 7f5d377fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.162+0000 7f5d377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.162+0000 7f5d377fe640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:20:10.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.162+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d501397c0 con 0x7f5d5013f710 2024-03-25T18:20:10.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.163+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 930001088 0 0) 0x7f5d3c004eb0 con 0x7f5d501430c0 2024-03-25T18:20:10.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.163+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d50138ab0 con 0x7f5d501430c0 2024-03-25T18:20:10.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.163+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5d48001ea0 con 0x7f5d501430c0 2024-03-25T18:20:10.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.163+0000 7f5d377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:10.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.163+0000 7f5d377fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:10.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.163+0000 7f5d377fe640 10 monclient(hunting): dump: 2024-03-25T18:20:10.165 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:10.165 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:10.165 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:10.165 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:10.165 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:10.165 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:10.165 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:10.165 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:10.165 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:10.165 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:10.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.163+0000 7f5d377fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1324894292 0 0) 0x7f5d50138ab0 con 0x7f5d501430c0 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 >> v1:172.21.15.122:6790/0 conn(0x7f5d5013f710 legacy=0x7f5d501383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 >> v1:172.21.15.122:6789/0 conn(0x7f5d500a98f0 legacy=0x7f5d501d7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d377fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d377fe640 1 monclient: found mon.b 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d377fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d50138c40 con 0x7f5d501430c0 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d377fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d377fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d55457640 5 monclient: authenticate success, global_id 4396 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d55457640 10 monclient: _renew_subs 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d55457640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d55457640 1 -- 172.21.15.122:0/3760265981 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d50137690 con 0x7f5d501430c0 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d55457640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d55457640 10 monclient: _renew_subs 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d55457640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d55457640 1 -- 172.21.15.122:0/3760265981 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d50134900 con 0x7f5d501430c0 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.169+0000 7f5d55457640 1 librados: init done 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.170+0000 7f5d55457640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.170+0000 7f5d55457640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.170+0000 7f5d55457640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.170+0000 7f5d55457640 1 -- 172.21.15.122:0/3760265981 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d501d79a0 con 0x7f5d501430c0 2024-03-25T18:20:10.455 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.453+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d48003c20 con 0x7f5d501430c0 2024-03-25T18:20:10.455 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.453+0000 7f5d377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:10.455 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.453+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5d48002100 con 0x7f5d501430c0 2024-03-25T18:20:10.455 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.453+0000 7f5d377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:10.455 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.453+0000 7f5d377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:10.455 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.453+0000 7f5d377fe640 10 monclient: dump: 2024-03-25T18:20:10.455 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:10.455 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:10.455 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:10.455 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:10.455 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:10.455 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:10.455 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:10.456 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:10.456 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:10.456 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:10.456 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.453+0000 7f5d4d7fa640 4 set_mon_vals no callback set 2024-03-25T18:20:10.456 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.454+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f5d48004070 con 0x7f5d501430c0 2024-03-25T18:20:10.456 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.455+0000 7f5d4effd640 10 monclient: get_auth_request con 0x7f5d3c06de20 auth_method 0 2024-03-25T18:20:10.456 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.455+0000 7f5d377fe640 10 client.4396.objecter ms_handle_connect 0x7f5d3c06de20 2024-03-25T18:20:10.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.459+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f5d48054c70 con 0x7f5d501430c0 2024-03-25T18:20:10.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.459+0000 7f5d377fe640 10 client.4396.objecter ms_dispatch 0x7f5d50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:10.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.459+0000 7f5d377fe640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:10.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.459+0000 7f5d377fe640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:10.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.459+0000 7f5d377fe640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-25T18:20:10.461 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.459+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d48050780 con 0x7f5d501430c0 2024-03-25T18:20:10.461 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.459+0000 7f5d377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:10.461 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.459+0000 7f5d377fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:10.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.700+0000 7f5d55457640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:10.702 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.700+0000 7f5d55457640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:10.702 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.700+0000 7f5d55457640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:10.702 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.700+0000 7f5d55457640 1 -- 172.21.15.122:0/3760265981 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d500b3ef0 con 0x7f5d501430c0 2024-03-25T18:20:10.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.812+0000 7f5d377fe640 1 -- 172.21.15.122:0/3760265981 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f5d480526b0 con 0x7f5d501430c0 2024-03-25T18:20:10.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.812+0000 7f5d377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:10.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.812+0000 7f5d377fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:10.814 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:20:10.814 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:10.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.814+0000 7f5d55457640 10 librados: watch_flush enter 2024-03-25T18:20:10.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.814+0000 7f5d55457640 10 librados: watch_flush exit 2024-03-25T18:20:10.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.814+0000 7f5d55457640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-25T18:20:10.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.814+0000 7f5d55457640 10 client.4396.objecter successfully canceled tick 2024-03-25T18:20:10.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.814+0000 7f5d55457640 1 -- 172.21.15.122:0/3760265981 >> v1:172.21.15.204:6800/676228005 conn(0x7f5d3c06de20 legacy=0x7f5d3c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.814+0000 7f5d55457640 10 monclient: shutdown 2024-03-25T18:20:10.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.814+0000 7f5d55457640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:10.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.814+0000 7f5d55457640 1 -- 172.21.15.122:0/3760265981 >> v1:172.21.15.204:6789/0 conn(0x7f5d501430c0 legacy=0x7f5d50134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.814+0000 7f5d55457640 1 -- 172.21.15.122:0/3760265981 shutdown_connections 2024-03-25T18:20:10.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.814+0000 7f5d55457640 1 -- 172.21.15.122:0/3760265981 >> 172.21.15.122:0/3760265981 conn(0x7f5d500b7ea0 msgr2=0x7f5d5013d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:10.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.815+0000 7f5d55457640 1 -- 172.21.15.122:0/3760265981 shutdown_connections 2024-03-25T18:20:10.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.815+0000 7f5d55457640 1 -- 172.21.15.122:0/3760265981 wait complete. 2024-03-25T18:20:10.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:10.815+0000 7f5d55457640 1 librados: shutdown 2024-03-25T18:20:10.826 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:17.633 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:17.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.747+0000 7f01d8a81640 10 monclient: get_monmap_and_config 2024-03-25T18:20:17.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.747+0000 7f01d8a81640 10 monclient: build_initial_monmap 2024-03-25T18:20:17.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.747+0000 7f01d8a81640 10 monclient: monmap: 2024-03-25T18:20:17.749 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:20:17.749 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:17.749 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:20:17.748740+0000 2024-03-25T18:20:17.749 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:20:17.748740+0000 2024-03-25T18:20:17.749 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:17.749 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:17.749 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:20:17.749 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:20:17.749 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:20:17.749 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:17.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.748+0000 7f01d8a81640 1 Processor -- start 2024-03-25T18:20:17.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.748+0000 7f01d8a81640 1 -- start start 2024-03-25T18:20:17.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.748+0000 7f01d8a81640 10 monclient: init 2024-03-25T18:20:17.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.749+0000 7f01d8a81640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:17.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.749+0000 7f01d8a81640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:17.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.749+0000 7f01d8a81640 10 monclient(hunting): picked mon.noname-b con 0x7f01d40a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:20:17.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.749+0000 7f01d8a81640 10 monclient(hunting): picked mon.noname-a con 0x7f01d413f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:20:17.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.749+0000 7f01d8a81640 10 monclient(hunting): picked mon.noname-c con 0x7f01d41430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:20:17.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.749+0000 7f01d8a81640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d4142520 con 0x7f01d413f710 2024-03-25T18:20:17.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.749+0000 7f01d8a81640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d40b28a0 con 0x7f01d41430c0 2024-03-25T18:20:17.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.749+0000 7f01d8a81640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d40b2a30 con 0x7f01d40a98f0 2024-03-25T18:20:17.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.749+0000 7f01d8a81640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:17.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.749+0000 7f01d8a81640 10 monclient(hunting): authenticate will time out at 7541.015231s 2024-03-25T18:20:17.758 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.757+0000 7f01d1d9b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f01d413f710 0x7f01d41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:49582/0 (socket says 172.21.15.122:49582) 2024-03-25T18:20:17.758 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.757+0000 7f01d1d9b640 1 -- 172.21.15.122:0/2969653644 learned_addr learned my addr 172.21.15.122:0/2969653644 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:20:17.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.778+0000 7f01d159a640 1 -- 172.21.15.122:0/2969653644 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3879051701 0 0) 0x7f01d40b28a0 con 0x7f01d41430c0 2024-03-25T18:20:17.780 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.778+0000 7f01d159a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.780 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.778+0000 7f01d159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.780 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.778+0000 7f01d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.780 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.778+0000 7f01d159a640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:20:17.780 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.778+0000 7f01d159a640 1 -- 172.21.15.122:0/2969653644 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01bc003520 con 0x7f01d41430c0 2024-03-25T18:20:17.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.788+0000 7f01d159a640 1 -- 172.21.15.122:0/2969653644 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858070130 0 0) 0x7f01d40b2a30 con 0x7f01d40a98f0 2024-03-25T18:20:17.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.788+0000 7f01d159a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.788+0000 7f01d159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.788+0000 7f01d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.788+0000 7f01d159a640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:20:17.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.788+0000 7f01d159a640 1 -- 172.21.15.122:0/2969653644 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01d40b28a0 con 0x7f01d40a98f0 2024-03-25T18:20:17.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.793+0000 7f01d159a640 1 -- 172.21.15.122:0/2969653644 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235970695 0 0) 0x7f01d4142520 con 0x7f01d413f710 2024-03-25T18:20:17.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.793+0000 7f01d159a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.793+0000 7f01d159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.793+0000 7f01d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.793+0000 7f01d159a640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:20:17.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.793+0000 7f01d159a640 1 -- 172.21.15.122:0/2969653644 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01d40b2a30 con 0x7f01d413f710 2024-03-25T18:20:17.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.805+0000 7f01d159a640 1 -- 172.21.15.122:0/2969653644 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 731785863 0 0) 0x7f01bc003520 con 0x7f01d41430c0 2024-03-25T18:20:17.807 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.805+0000 7f01d159a640 1 -- 172.21.15.122:0/2969653644 >> v1:172.21.15.122:6790/0 conn(0x7f01d40a98f0 legacy=0x7f01d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.807 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.805+0000 7f01d159a640 1 -- 172.21.15.122:0/2969653644 >> v1:172.21.15.122:6789/0 conn(0x7f01d413f710 legacy=0x7f01d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.807 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.805+0000 7f01d159a640 10 monclient: _finish_hunting 0 2024-03-25T18:20:17.807 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.805+0000 7f01d159a640 1 monclient: found mon.noname-c 2024-03-25T18:20:17.807 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.805+0000 7f01d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:17.807 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.805+0000 7f01d159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:20:17.807 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.805+0000 7f01d159a640 1 -- 172.21.15.122:0/2969653644 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d40b44c0 con 0x7f01d41430c0 2024-03-25T18:20:17.807 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.805+0000 7f01d159a640 10 monclient: _finish_auth 0 2024-03-25T18:20:17.807 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.805+0000 7f01d159a640 10 monclient: _check_auth_tickets 2024-03-25T18:20:17.807 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.805+0000 7f01d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:17.807 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.805+0000 7f01d159a640 1 -- 172.21.15.122:0/2969653644 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f01cc001a10 con 0x7f01d41430c0 2024-03-25T18:20:17.807 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.805+0000 7f01d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.805+0000 7f01d159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:17.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.805+0000 7f01d159a640 10 monclient: dump: 2024-03-25T18:20:17.808 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:17.808 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:17.808 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:17.808 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:17.808 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.808 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:17.808 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:17.808 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:17.808 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:17.808 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:17.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.805+0000 7f01d8a81640 5 monclient: authenticate success, global_id 4399 2024-03-25T18:20:17.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.805+0000 7f01d8a81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:17.842 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.841+0000 7f01d159a640 1 -- 172.21.15.122:0/2969653644 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01cc001b70 con 0x7f01d41430c0 2024-03-25T18:20:17.842 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.841+0000 7f01d159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:17.842 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.841+0000 7f01d159a640 1 -- 172.21.15.122:0/2969653644 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f01cc001f80 con 0x7f01d41430c0 2024-03-25T18:20:17.842 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.841+0000 7f01d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.843 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.841+0000 7f01d159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:17.843 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.841+0000 7f01d159a640 10 monclient: dump: 2024-03-25T18:20:17.843 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:17.843 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:17.843 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:17.843 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:17.843 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.843 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:17.843 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:17.843 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:17.843 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:17.844 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:17.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.841+0000 7f01d8a81640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:17.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.841+0000 7f01d8a81640 4 set_mon_vals no callback set 2024-03-25T18:20:17.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.842+0000 7f01d8a81640 10 monclient: shutdown 2024-03-25T18:20:17.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.842+0000 7f01d8a81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:17.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.842+0000 7f01d8a81640 1 -- 172.21.15.122:0/2969653644 >> v1:172.21.15.204:6789/0 conn(0x7f01d41430c0 legacy=0x7f01d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.842+0000 7f01d8a81640 1 -- 172.21.15.122:0/2969653644 shutdown_connections 2024-03-25T18:20:17.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.842+0000 7f01d8a81640 1 -- 172.21.15.122:0/2969653644 >> 172.21.15.122:0/2969653644 conn(0x7f01d40b7ea0 msgr2=0x7f01d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:17.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.842+0000 7f01d8a81640 1 -- 172.21.15.122:0/2969653644 shutdown_connections 2024-03-25T18:20:17.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.842+0000 7f01d8a81640 1 -- 172.21.15.122:0/2969653644 wait complete. 2024-03-25T18:20:17.847 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.843+0000 7f01d8a81640 10 monclient: build_initial_monmap 2024-03-25T18:20:17.847 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.843+0000 7f01d8a81640 10 monclient: monmap: 2024-03-25T18:20:17.847 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:20:17.847 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:17.847 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:20:17.847 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:20:17.847 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:17.847 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:20:17.847 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:20:17.847 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:20:17.847 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:20:17.847 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:17.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.843+0000 7f01d8a81640 1 librados: starting msgr at 2024-03-25T18:20:17.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.843+0000 7f01d8a81640 1 librados: starting objecter 2024-03-25T18:20:17.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.843+0000 7f01d8a81640 1 Processor -- start 2024-03-25T18:20:17.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.843+0000 7f01d8a81640 1 -- start start 2024-03-25T18:20:17.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.843+0000 7f01d8a81640 1 librados: setting wanted keys 2024-03-25T18:20:17.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.843+0000 7f01d8a81640 1 librados: calling monclient init 2024-03-25T18:20:17.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.843+0000 7f01d8a81640 10 monclient: init 2024-03-25T18:20:17.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.844+0000 7f01d8a81640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:17.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.844+0000 7f01d8a81640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:17.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.844+0000 7f01d8a81640 10 monclient(hunting): picked mon.noname-b con 0x7f01d40a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:20:17.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.844+0000 7f01d8a81640 10 monclient(hunting): picked mon.noname-a con 0x7f01d413f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:20:17.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.844+0000 7f01d8a81640 10 monclient(hunting): picked mon.noname-c con 0x7f01d41430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:20:17.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.844+0000 7f01d8a81640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d4137190 con 0x7f01d413f710 2024-03-25T18:20:17.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.844+0000 7f01d8a81640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d41d41e0 con 0x7f01d40a98f0 2024-03-25T18:20:17.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.844+0000 7f01d8a81640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01d41d4430 con 0x7f01d41430c0 2024-03-25T18:20:17.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.844+0000 7f01d8a81640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:17.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.844+0000 7f01d8a81640 10 monclient(hunting): authenticate will time out at 7541.110565s 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.854+0000 7f01d259c640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f01d40a98f0 0x7f01d41d4950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:53990/0 (socket says 172.21.15.122:53990) 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.854+0000 7f01d259c640 1 -- 172.21.15.122:0/987527262 learned_addr learned my addr 172.21.15.122:0/987527262 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:20:17.856 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.855+0000 7f01c2ffd640 10 client.?.objecter ms_handle_connect 0x7f01d40a98f0 2024-03-25T18:20:17.856 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.855+0000 7f01c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:17.857 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.856+0000 7f01c2ffd640 10 client.?.objecter ms_handle_connect 0x7f01d413f710 2024-03-25T18:20:17.857 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.856+0000 7f01c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:17.863 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.862+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444298588 0 0) 0x7f01d41d41e0 con 0x7f01d40a98f0 2024-03-25T18:20:17.863 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.862+0000 7f01c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.863 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.862+0000 7f01c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.863 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.862+0000 7f01c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.863 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.862+0000 7f01c2ffd640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:20:17.863 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.862+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01b4004eb0 con 0x7f01d40a98f0 2024-03-25T18:20:17.868 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.867+0000 7f01c2ffd640 10 client.?.objecter ms_handle_connect 0x7f01d41430c0 2024-03-25T18:20:17.868 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.867+0000 7f01c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:17.872 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.871+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2527385743 0 0) 0x7f01d4137190 con 0x7f01d413f710 2024-03-25T18:20:17.872 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.871+0000 7f01c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.872 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.871+0000 7f01c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.872 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.871+0000 7f01c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.872 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.871+0000 7f01c2ffd640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:20:17.872 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.871+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01d41d41e0 con 0x7f01d413f710 2024-03-25T18:20:17.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.872+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 353110085 0 0) 0x7f01b4004eb0 con 0x7f01d40a98f0 2024-03-25T18:20:17.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.872+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01d4137190 con 0x7f01d40a98f0 2024-03-25T18:20:17.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.872+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f01c4001a30 con 0x7f01d40a98f0 2024-03-25T18:20:17.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.872+0000 7f01c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.872+0000 7f01c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:17.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.872+0000 7f01c2ffd640 10 monclient(hunting): dump: 2024-03-25T18:20:17.874 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:17.874 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:17.874 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:17.874 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:17.874 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.874 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:17.874 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:17.874 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:17.874 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:17.874 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:17.874 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.872+0000 7f01c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:17.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.889+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 293225942 0 0) 0x7f01d41d4430 con 0x7f01d41430c0 2024-03-25T18:20:17.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.889+0000 7f01c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.889+0000 7f01c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.889+0000 7f01c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.889+0000 7f01c2ffd640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:20:17.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.889+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01b4004eb0 con 0x7f01d41430c0 2024-03-25T18:20:17.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.889+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1444699059 0 0) 0x7f01d41d41e0 con 0x7f01d413f710 2024-03-25T18:20:17.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.889+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01d41d4430 con 0x7f01d413f710 2024-03-25T18:20:17.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.889+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f01c8001870 con 0x7f01d413f710 2024-03-25T18:20:17.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.889+0000 7f01c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.889+0000 7f01c2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:17.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.889+0000 7f01c2ffd640 10 monclient(hunting): dump: 2024-03-25T18:20:17.892 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:17.893 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:17.893 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:17.893 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:17.893 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.893 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:17.893 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:17.893 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:17.893 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:17.893 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:17.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2744531126 0 0) 0x7f01d4137190 con 0x7f01d40a98f0 2024-03-25T18:20:17.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 >> v1:172.21.15.122:6790/0 conn(0x7f01d41430c0 legacy=0x7f01d4130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 >> v1:172.21.15.122:6789/0 conn(0x7f01d413f710 legacy=0x7f01d41352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01c2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:17.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01c2ffd640 1 monclient: found mon.b 2024-03-25T18:20:17.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:17.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01c2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:17.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d413bf90 con 0x7f01d40a98f0 2024-03-25T18:20:17.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01c2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:20:17.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01c2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:17.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:17.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01d8a81640 5 monclient: authenticate success, global_id 4402 2024-03-25T18:20:17.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01d8a81640 10 monclient: _renew_subs 2024-03-25T18:20:17.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01d8a81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:17.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01d8a81640 1 -- 172.21.15.122:0/987527262 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d41d5060 con 0x7f01d40a98f0 2024-03-25T18:20:17.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01d8a81640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:17.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01d8a81640 10 monclient: _renew_subs 2024-03-25T18:20:17.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01d8a81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:17.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01d8a81640 1 -- 172.21.15.122:0/987527262 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d4139f00 con 0x7f01d40a98f0 2024-03-25T18:20:17.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.890+0000 7f01d8a81640 1 librados: init done 2024-03-25T18:20:17.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.891+0000 7f01d8a81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:17.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.891+0000 7f01d8a81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:17.896 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.891+0000 7f01d8a81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:17.896 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:17.891+0000 7f01d8a81640 1 -- 172.21.15.122:0/987527262 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d41d45c0 con 0x7f01d40a98f0 2024-03-25T18:20:18.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.179+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01c4001c70 con 0x7f01d40a98f0 2024-03-25T18:20:18.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.179+0000 7f01c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:18.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.179+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f01c4003a60 con 0x7f01d40a98f0 2024-03-25T18:20:18.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.179+0000 7f01c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:18.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.179+0000 7f01c2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:18.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.179+0000 7f01c2ffd640 10 monclient: dump: 2024-03-25T18:20:18.181 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:18.181 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:18.181 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:18.181 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:18.181 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:18.181 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:18.181 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:18.181 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:18.181 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:18.182 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:18.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.179+0000 7f01d0d99640 4 set_mon_vals no callback set 2024-03-25T18:20:18.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.180+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f01c4003c40 con 0x7f01d40a98f0 2024-03-25T18:20:18.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.180+0000 7f01d1d9b640 10 monclient: get_auth_request con 0x7f01b406dfb0 auth_method 0 2024-03-25T18:20:18.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.181+0000 7f01c2ffd640 10 client.4402.objecter ms_handle_connect 0x7f01b406dfb0 2024-03-25T18:20:18.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.185+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f01c40548d0 con 0x7f01d40a98f0 2024-03-25T18:20:18.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.185+0000 7f01c2ffd640 10 client.4402.objecter ms_dispatch 0x7f01d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:18.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.185+0000 7f01c2ffd640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:18.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.185+0000 7f01c2ffd640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:18.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.186+0000 7f01c2ffd640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-25T18:20:18.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.186+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f01c40503e0 con 0x7f01d40a98f0 2024-03-25T18:20:18.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.187+0000 7f01c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:18.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.187+0000 7f01c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:18.441 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.439+0000 7f01d8a81640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:18.441 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.439+0000 7f01d8a81640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:18.441 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.439+0000 7f01d8a81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:18.441 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.439+0000 7f01d8a81640 1 -- 172.21.15.122:0/987527262 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01d40627b0 con 0x7f01d40a98f0 2024-03-25T18:20:18.461 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.460+0000 7f01c2ffd640 1 -- 172.21.15.122:0/987527262 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f01c4052310 con 0x7f01d40a98f0 2024-03-25T18:20:18.461 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.460+0000 7f01c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:18.461 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.460+0000 7f01c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:18.462 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:20:18.462 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:18.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.462+0000 7f01d8a81640 10 librados: watch_flush enter 2024-03-25T18:20:18.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.462+0000 7f01d8a81640 10 librados: watch_flush exit 2024-03-25T18:20:18.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.462+0000 7f01d8a81640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-25T18:20:18.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.462+0000 7f01d8a81640 10 client.4402.objecter successfully canceled tick 2024-03-25T18:20:18.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.463+0000 7f01d8a81640 1 -- 172.21.15.122:0/987527262 >> v1:172.21.15.204:6800/676228005 conn(0x7f01b406dfb0 legacy=0x7f01b4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:18.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.463+0000 7f01d8a81640 10 monclient: shutdown 2024-03-25T18:20:18.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.463+0000 7f01d8a81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:18.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.463+0000 7f01d8a81640 1 -- 172.21.15.122:0/987527262 >> v1:172.21.15.204:6789/0 conn(0x7f01d40a98f0 legacy=0x7f01d41d4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:18.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.463+0000 7f01d8a81640 1 -- 172.21.15.122:0/987527262 shutdown_connections 2024-03-25T18:20:18.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.463+0000 7f01d8a81640 1 -- 172.21.15.122:0/987527262 >> 172.21.15.122:0/987527262 conn(0x7f01d40b7ea0 msgr2=0x7f01d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:18.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.463+0000 7f01d8a81640 1 -- 172.21.15.122:0/987527262 shutdown_connections 2024-03-25T18:20:18.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.463+0000 7f01d8a81640 1 -- 172.21.15.122:0/987527262 wait complete. 2024-03-25T18:20:18.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:18.463+0000 7f01d8a81640 1 librados: shutdown 2024-03-25T18:20:18.474 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:25.281 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:25.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.395+0000 7f0d724f1640 10 monclient: get_monmap_and_config 2024-03-25T18:20:25.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.395+0000 7f0d724f1640 10 monclient: build_initial_monmap 2024-03-25T18:20:25.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.395+0000 7f0d724f1640 10 monclient: monmap: 2024-03-25T18:20:25.397 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:20:25.397 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:25.397 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:20:25.396736+0000 2024-03-25T18:20:25.397 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:20:25.396736+0000 2024-03-25T18:20:25.397 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:25.397 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:25.397 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:20:25.397 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:20:25.397 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:20:25.397 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:25.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.396+0000 7f0d724f1640 1 Processor -- start 2024-03-25T18:20:25.398 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.396+0000 7f0d724f1640 1 -- start start 2024-03-25T18:20:25.398 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.396+0000 7f0d724f1640 10 monclient: init 2024-03-25T18:20:25.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.397+0000 7f0d724f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:25.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.397+0000 7f0d724f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:25.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.397+0000 7f0d724f1640 10 monclient(hunting): picked mon.noname-b con 0x7f0d6c0a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:20:25.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.397+0000 7f0d724f1640 10 monclient(hunting): picked mon.noname-c con 0x7f0d6c13f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:20:25.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.397+0000 7f0d724f1640 10 monclient(hunting): picked mon.noname-a con 0x7f0d6c1430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:20:25.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.398+0000 7f0d724f1640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6c142520 con 0x7f0d6c1430c0 2024-03-25T18:20:25.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.398+0000 7f0d724f1640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6c0b28a0 con 0x7f0d6c13f710 2024-03-25T18:20:25.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.398+0000 7f0d724f1640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6c0b2a30 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.398+0000 7f0d724f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:25.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.398+0000 7f0d724f1640 10 monclient(hunting): authenticate will time out at 7548.664049s 2024-03-25T18:20:25.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.403+0000 7f0d70a8e640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0d6c1430c0 0x7f0d6c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33878/0 (socket says 172.21.15.122:33878) 2024-03-25T18:20:25.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.403+0000 7f0d70a8e640 1 -- 172.21.15.122:0/4266006899 learned_addr learned my addr 172.21.15.122:0/4266006899 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:20:25.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.419+0000 7f0d6affd640 1 -- 172.21.15.122:0/4266006899 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972123522 0 0) 0x7f0d6c142520 con 0x7f0d6c1430c0 2024-03-25T18:20:25.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.419+0000 7f0d6affd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:25.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.419+0000 7f0d6affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:25.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.419+0000 7f0d6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:25.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.419+0000 7f0d6affd640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:20:25.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.419+0000 7f0d6affd640 1 -- 172.21.15.122:0/4266006899 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d54003520 con 0x7f0d6c1430c0 2024-03-25T18:20:25.431 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.430+0000 7f0d6affd640 1 -- 172.21.15.122:0/4266006899 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478611676 0 0) 0x7f0d6c0b28a0 con 0x7f0d6c13f710 2024-03-25T18:20:25.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.430+0000 7f0d6affd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:25.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.430+0000 7f0d6affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:25.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.430+0000 7f0d6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:25.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.430+0000 7f0d6affd640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:20:25.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.430+0000 7f0d6affd640 1 -- 172.21.15.122:0/4266006899 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d6c142520 con 0x7f0d6c13f710 2024-03-25T18:20:25.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.430+0000 7f0d6affd640 1 -- 172.21.15.122:0/4266006899 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106563181 0 0) 0x7f0d6c0b2a30 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.430+0000 7f0d6affd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:25.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.430+0000 7f0d6affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:25.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.430+0000 7f0d6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:25.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.430+0000 7f0d6affd640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:20:25.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.430+0000 7f0d6affd640 1 -- 172.21.15.122:0/4266006899 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d6c0b28a0 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.444 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d6affd640 1 -- 172.21.15.122:0/4266006899 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1257996778 0 0) 0x7f0d54003520 con 0x7f0d6c1430c0 2024-03-25T18:20:25.444 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d6affd640 1 -- 172.21.15.122:0/4266006899 >> v1:172.21.15.122:6790/0 conn(0x7f0d6c0a98f0 legacy=0x7f0d6c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.444 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d6affd640 1 -- 172.21.15.122:0/4266006899 >> v1:172.21.15.204:6789/0 conn(0x7f0d6c13f710 legacy=0x7f0d6c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d6affd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:25.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d6affd640 1 monclient: found mon.noname-a 2024-03-25T18:20:25.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:25.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d6affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:20:25.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d6affd640 1 -- 172.21.15.122:0/4266006899 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c0b44c0 con 0x7f0d6c1430c0 2024-03-25T18:20:25.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d6affd640 10 monclient: _finish_auth 0 2024-03-25T18:20:25.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d6affd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:25.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:25.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d6affd640 1 -- 172.21.15.122:0/4266006899 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0d64001a10 con 0x7f0d6c1430c0 2024-03-25T18:20:25.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d724f1640 5 monclient: authenticate success, global_id 4407 2024-03-25T18:20:25.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d724f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:25.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:25.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d6affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:25.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d6affd640 10 monclient: dump: 2024-03-25T18:20:25.445 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:25.445 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:25.446 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:25.446 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:25.446 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:25.446 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:25.446 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:25.446 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:25.446 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:25.446 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:25.446 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.443+0000 7f0d724f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:25.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.464+0000 7f0d6affd640 1 -- 172.21.15.122:0/4266006899 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d64001b70 con 0x7f0d6c1430c0 2024-03-25T18:20:25.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.464+0000 7f0d6affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:25.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.464+0000 7f0d6affd640 1 -- 172.21.15.122:0/4266006899 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0d64001f80 con 0x7f0d6c1430c0 2024-03-25T18:20:25.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.464+0000 7f0d6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:25.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.464+0000 7f0d6affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:25.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.464+0000 7f0d6affd640 10 monclient: dump: 2024-03-25T18:20:25.465 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:25.465 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:25.465 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:25.465 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:25.465 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:25.465 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:25.466 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:25.466 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:25.466 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:25.466 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:25.466 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.464+0000 7f0d724f1640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:25.466 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.464+0000 7f0d724f1640 4 set_mon_vals no callback set 2024-03-25T18:20:25.467 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.464+0000 7f0d724f1640 10 monclient: shutdown 2024-03-25T18:20:25.467 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.464+0000 7f0d724f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:25.467 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.464+0000 7f0d724f1640 1 -- 172.21.15.122:0/4266006899 >> v1:172.21.15.122:6789/0 conn(0x7f0d6c1430c0 legacy=0x7f0d6c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.467 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.464+0000 7f0d724f1640 1 -- 172.21.15.122:0/4266006899 shutdown_connections 2024-03-25T18:20:25.467 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.464+0000 7f0d724f1640 1 -- 172.21.15.122:0/4266006899 >> 172.21.15.122:0/4266006899 conn(0x7f0d6c0b7ea0 msgr2=0x7f0d6c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:25.467 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.464+0000 7f0d724f1640 1 -- 172.21.15.122:0/4266006899 shutdown_connections 2024-03-25T18:20:25.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.464+0000 7f0d724f1640 1 -- 172.21.15.122:0/4266006899 wait complete. 2024-03-25T18:20:25.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.465+0000 7f0d724f1640 10 monclient: build_initial_monmap 2024-03-25T18:20:25.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.465+0000 7f0d724f1640 10 monclient: monmap: 2024-03-25T18:20:25.468 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:20:25.468 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:25.468 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:20:25.468 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:20:25.468 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:25.468 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:20:25.468 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:20:25.468 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:20:25.468 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:20:25.468 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:25.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.465+0000 7f0d724f1640 1 librados: starting msgr at 2024-03-25T18:20:25.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.465+0000 7f0d724f1640 1 librados: starting objecter 2024-03-25T18:20:25.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.466+0000 7f0d724f1640 1 Processor -- start 2024-03-25T18:20:25.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.466+0000 7f0d724f1640 1 -- start start 2024-03-25T18:20:25.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.466+0000 7f0d724f1640 1 librados: setting wanted keys 2024-03-25T18:20:25.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.466+0000 7f0d724f1640 1 librados: calling monclient init 2024-03-25T18:20:25.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.466+0000 7f0d724f1640 10 monclient: init 2024-03-25T18:20:25.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.466+0000 7f0d724f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:25.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.466+0000 7f0d724f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:25.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.467+0000 7f0d724f1640 10 monclient(hunting): picked mon.noname-b con 0x7f0d6c0a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:20:25.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.467+0000 7f0d724f1640 10 monclient(hunting): picked mon.noname-c con 0x7f0d6c13f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:20:25.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.467+0000 7f0d724f1640 10 monclient(hunting): picked mon.noname-a con 0x7f0d6c1430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:20:25.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.467+0000 7f0d724f1640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6c136e20 con 0x7f0d6c1430c0 2024-03-25T18:20:25.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.467+0000 7f0d724f1640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6c0b7940 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.467+0000 7f0d724f1640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6c1351a0 con 0x7f0d6c13f710 2024-03-25T18:20:25.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.467+0000 7f0d724f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:25.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.467+0000 7f0d724f1640 10 monclient(hunting): authenticate will time out at 7548.733633s 2024-03-25T18:20:25.475 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.474+0000 7f0d70a8e640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f0d6c1430c0 0x7f0d6c1e1fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:33936/0 (socket says 172.21.15.122:33936) 2024-03-25T18:20:25.476 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.474+0000 7f0d70a8e640 1 -- 172.21.15.122:0/2969343432 learned_addr learned my addr 172.21.15.122:0/2969343432 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:20:25.476 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.475+0000 7f0d68ff9640 10 client.?.objecter ms_handle_connect 0x7f0d6c1430c0 2024-03-25T18:20:25.476 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.475+0000 7f0d68ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:25.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.477+0000 7f0d68ff9640 10 client.?.objecter ms_handle_connect 0x7f0d6c13f710 2024-03-25T18:20:25.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.477+0000 7f0d68ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:25.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.478+0000 7f0d68ff9640 10 client.?.objecter ms_handle_connect 0x7f0d6c0a98f0 2024-03-25T18:20:25.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.478+0000 7f0d68ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:25.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.491+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576924948 0 0) 0x7f0d6c136e20 con 0x7f0d6c1430c0 2024-03-25T18:20:25.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.491+0000 7f0d68ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:25.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.491+0000 7f0d68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:25.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.491+0000 7f0d68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:25.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.491+0000 7f0d68ff9640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:20:25.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.491+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d58004eb0 con 0x7f0d6c1430c0 2024-03-25T18:20:25.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.492+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715018272 0 0) 0x7f0d6c0b7940 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.492+0000 7f0d68ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:25.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.492+0000 7f0d68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:25.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.492+0000 7f0d68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:25.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.492+0000 7f0d68ff9640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:20:25.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.492+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d6c136e20 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.495+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633455668 0 0) 0x7f0d6c1351a0 con 0x7f0d6c13f710 2024-03-25T18:20:25.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.495+0000 7f0d68ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:20:25.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.495+0000 7f0d68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:25.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.495+0000 7f0d68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:25.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.495+0000 7f0d68ff9640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:20:25.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.495+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d6c0b7940 con 0x7f0d6c13f710 2024-03-25T18:20:25.504 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.502+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 510126662 0 0) 0x7f0d6c136e20 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.504 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.502+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d6c1351a0 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.504 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.502+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0d5c001870 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.504 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.502+0000 7f0d68ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:25.504 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.502+0000 7f0d68ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:25.504 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.502+0000 7f0d68ff9640 10 monclient(hunting): dump: 2024-03-25T18:20:25.504 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:25.504 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:25.504 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:25.504 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:25.505 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:25.505 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:25.505 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:25.505 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:25.505 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:25.505 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:25.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.502+0000 7f0d68ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:25.512 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.511+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1357355469 0 0) 0x7f0d58004eb0 con 0x7f0d6c1430c0 2024-03-25T18:20:25.512 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.511+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d6c136e20 con 0x7f0d6c1430c0 2024-03-25T18:20:25.512 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.511+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0d64001ec0 con 0x7f0d6c1430c0 2024-03-25T18:20:25.512 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.511+0000 7f0d68ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:25.512 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.511+0000 7f0d68ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:25.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.511+0000 7f0d68ff9640 10 monclient(hunting): dump: 2024-03-25T18:20:25.513 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:25.513 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:25.513 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:25.513 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:25.513 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:25.513 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:25.513 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:25.513 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:25.513 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:25.513 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 193354987 0 0) 0x7f0d6c1351a0 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 >> v1:172.21.15.122:6790/0 conn(0x7f0d6c13f710 legacy=0x7f0d6c1db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 >> v1:172.21.15.122:6789/0 conn(0x7f0d6c1430c0 legacy=0x7f0d6c1e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d68ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:20:25.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d68ff9640 1 monclient: found mon.b 2024-03-25T18:20:25.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d68ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:25.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d68ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:25.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c135330 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d68ff9640 10 monclient: _finish_auth 0 2024-03-25T18:20:25.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d68ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:20:25.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:25.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d724f1640 5 monclient: authenticate success, global_id 4408 2024-03-25T18:20:25.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d724f1640 10 monclient: _renew_subs 2024-03-25T18:20:25.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d724f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:25.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d724f1640 1 -- 172.21.15.122:0/2969343432 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6c135560 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d724f1640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:25.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d724f1640 10 monclient: _renew_subs 2024-03-25T18:20:25.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d724f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:25.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d724f1640 1 -- 172.21.15.122:0/2969343432 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6c13b490 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.517 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.514+0000 7f0d724f1640 1 librados: init done 2024-03-25T18:20:25.517 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.515+0000 7f0d724f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:25.517 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.515+0000 7f0d724f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:25.517 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.515+0000 7f0d724f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:25.517 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.515+0000 7f0d724f1640 1 -- 172.21.15.122:0/2969343432 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d6c0b7730 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.790+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d5c003610 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.790+0000 7f0d68ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:25.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.790+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0d5c001d80 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.790+0000 7f0d68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:25.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.790+0000 7f0d68ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:25.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.790+0000 7f0d68ff9640 10 monclient: dump: 2024-03-25T18:20:25.792 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:25.792 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:25.792 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:25.792 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:25.792 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:25.792 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:25.792 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:25.792 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:25.792 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:25.792 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:25.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.790+0000 7f0d6a7fc640 4 set_mon_vals no callback set 2024-03-25T18:20:25.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.791+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f0d5c003b20 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.792+0000 7f0d6b7fe640 10 monclient: get_auth_request con 0x7f0d5806df70 auth_method 0 2024-03-25T18:20:25.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.792+0000 7f0d68ff9640 10 client.4408.objecter ms_handle_connect 0x7f0d5806df70 2024-03-25T18:20:25.798 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.797+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f0d5c053570 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.798 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.797+0000 7f0d68ff9640 10 client.4408.objecter ms_dispatch 0x7f0d6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:25.798 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.797+0000 7f0d68ff9640 3 client.4408.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:25.799 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.797+0000 7f0d68ff9640 3 client.4408.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:25.799 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.797+0000 7f0d68ff9640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-25T18:20:25.799 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.798+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d5c04f080 con 0x7f0d6c0a98f0 2024-03-25T18:20:25.799 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.798+0000 7f0d68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:25.799 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:25.798+0000 7f0d68ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:26.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.050+0000 7f0d724f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:26.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.050+0000 7f0d724f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:26.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.050+0000 7f0d724f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:26.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.050+0000 7f0d724f1640 1 -- 172.21.15.122:0/2969343432 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d6c0627b0 con 0x7f0d6c0a98f0 2024-03-25T18:20:26.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.188+0000 7f0d68ff9640 1 -- 172.21.15.122:0/2969343432 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f0d5c050fb0 con 0x7f0d6c0a98f0 2024-03-25T18:20:26.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.188+0000 7f0d68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:26.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.188+0000 7f0d68ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:26.190 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:20:26.190 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:26.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.190+0000 7f0d724f1640 10 librados: watch_flush enter 2024-03-25T18:20:26.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.190+0000 7f0d724f1640 10 librados: watch_flush exit 2024-03-25T18:20:26.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.190+0000 7f0d724f1640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-25T18:20:26.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.190+0000 7f0d724f1640 10 client.4408.objecter successfully canceled tick 2024-03-25T18:20:26.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.190+0000 7f0d724f1640 1 -- 172.21.15.122:0/2969343432 >> v1:172.21.15.204:6800/676228005 conn(0x7f0d5806df70 legacy=0x7f0d58070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:26.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.190+0000 7f0d724f1640 10 monclient: shutdown 2024-03-25T18:20:26.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.190+0000 7f0d724f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:26.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.190+0000 7f0d724f1640 1 -- 172.21.15.122:0/2969343432 >> v1:172.21.15.204:6789/0 conn(0x7f0d6c0a98f0 legacy=0x7f0d6c1daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:26.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.190+0000 7f0d724f1640 1 -- 172.21.15.122:0/2969343432 shutdown_connections 2024-03-25T18:20:26.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.190+0000 7f0d724f1640 1 -- 172.21.15.122:0/2969343432 >> 172.21.15.122:0/2969343432 conn(0x7f0d6c0b7ea0 msgr2=0x7f0d6c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:26.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.191+0000 7f0d724f1640 1 -- 172.21.15.122:0/2969343432 shutdown_connections 2024-03-25T18:20:26.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.191+0000 7f0d724f1640 1 -- 172.21.15.122:0/2969343432 wait complete. 2024-03-25T18:20:26.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:26.191+0000 7f0d724f1640 1 librados: shutdown 2024-03-25T18:20:26.203 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:33.009 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:33.125 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.123+0000 7f7a848f6640 10 monclient: get_monmap_and_config 2024-03-25T18:20:33.125 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.123+0000 7f7a848f6640 10 monclient: build_initial_monmap 2024-03-25T18:20:33.125 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.124+0000 7f7a848f6640 10 monclient: monmap: 2024-03-25T18:20:33.125 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:20:33.125 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:33.125 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:20:33.125113+0000 2024-03-25T18:20:33.125 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:20:33.125113+0000 2024-03-25T18:20:33.125 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:33.125 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:33.125 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:20:33.125 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:20:33.125 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:20:33.126 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:33.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.124+0000 7f7a848f6640 1 Processor -- start 2024-03-25T18:20:33.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.124+0000 7f7a848f6640 1 -- start start 2024-03-25T18:20:33.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.124+0000 7f7a848f6640 10 monclient: init 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.125+0000 7f7a848f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.125+0000 7f7a848f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.125+0000 7f7a848f6640 10 monclient(hunting): picked mon.noname-c con 0x7f7a8013f990 addr v1:172.21.15.204:6789/0 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.125+0000 7f7a848f6640 10 monclient(hunting): picked mon.noname-a con 0x7f7a8013fd70 addr v1:172.21.15.122:6789/0 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.125+0000 7f7a848f6640 10 monclient(hunting): picked mon.noname-b con 0x7f7a800afa60 addr v1:172.21.15.122:6790/0 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.125+0000 7f7a848f6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a800afeb0 con 0x7f7a8013fd70 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.125+0000 7f7a848f6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a800b0100 con 0x7f7a8013f990 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.125+0000 7f7a848f6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a800b1150 con 0x7f7a800afa60 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.125+0000 7f7a848f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.125+0000 7f7a848f6640 10 monclient(hunting): authenticate will time out at 7556.391751s 2024-03-25T18:20:33.137 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.136+0000 7f7a7ed9d640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f7a800afa60 0x7f7a8014dbf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:52394/0 (socket says 172.21.15.122:52394) 2024-03-25T18:20:33.137 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.136+0000 7f7a7ed9d640 1 -- 172.21.15.122:0/2640923 learned_addr learned my addr 172.21.15.122:0/2640923 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:20:33.146 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.145+0000 7f7a7d59a640 1 -- 172.21.15.122:0/2640923 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792068828 0 0) 0x7f7a800afeb0 con 0x7f7a8013fd70 2024-03-25T18:20:33.146 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.145+0000 7f7a7d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:33.146 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.145+0000 7f7a7d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:33.146 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.145+0000 7f7a7d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:33.146 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.145+0000 7f7a7d59a640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:20:33.146 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.145+0000 7f7a7d59a640 1 -- 172.21.15.122:0/2640923 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a68003520 con 0x7f7a8013fd70 2024-03-25T18:20:33.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.149+0000 7f7a7d59a640 1 -- 172.21.15.122:0/2640923 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372559604 0 0) 0x7f7a800b1150 con 0x7f7a800afa60 2024-03-25T18:20:33.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.149+0000 7f7a7d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:33.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.149+0000 7f7a7d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:33.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.149+0000 7f7a7d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:33.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.149+0000 7f7a7d59a640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:20:33.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.149+0000 7f7a7d59a640 1 -- 172.21.15.122:0/2640923 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a800afeb0 con 0x7f7a800afa60 2024-03-25T18:20:33.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.150+0000 7f7a7d59a640 1 -- 172.21.15.122:0/2640923 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799434295 0 0) 0x7f7a800b0100 con 0x7f7a8013f990 2024-03-25T18:20:33.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.150+0000 7f7a7d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:33.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.150+0000 7f7a7d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:33.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.150+0000 7f7a7d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:33.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.150+0000 7f7a7d59a640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:20:33.152 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.150+0000 7f7a7d59a640 1 -- 172.21.15.122:0/2640923 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a800b1150 con 0x7f7a8013f990 2024-03-25T18:20:33.155 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.153+0000 7f7a7d59a640 1 -- 172.21.15.122:0/2640923 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2446223767 0 0) 0x7f7a68003520 con 0x7f7a8013fd70 2024-03-25T18:20:33.155 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.153+0000 7f7a7d59a640 1 -- 172.21.15.122:0/2640923 >> v1:172.21.15.122:6790/0 conn(0x7f7a800afa60 legacy=0x7f7a8014dbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.155 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.153+0000 7f7a7d59a640 1 -- 172.21.15.122:0/2640923 >> v1:172.21.15.204:6789/0 conn(0x7f7a8013f990 legacy=0x7f7a8006ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.155 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.153+0000 7f7a7d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:20:33.155 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.153+0000 7f7a7d59a640 1 monclient: found mon.noname-a 2024-03-25T18:20:33.155 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.153+0000 7f7a7d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.153+0000 7f7a7d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.153+0000 7f7a7d59a640 1 -- 172.21.15.122:0/2640923 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a800aede0 con 0x7f7a8013fd70 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.153+0000 7f7a7d59a640 10 monclient: _finish_auth 0 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.153+0000 7f7a7d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.153+0000 7f7a7d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.153+0000 7f7a7d59a640 1 -- 172.21.15.122:0/2640923 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f7a74001690 con 0x7f7a8013fd70 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.153+0000 7f7a7d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.153+0000 7f7a7d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.153+0000 7f7a7d59a640 10 monclient: dump: 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:33.157 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:33.157 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:33.157 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:33.157 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.153+0000 7f7a848f6640 5 monclient: authenticate success, global_id 4413 2024-03-25T18:20:33.157 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.153+0000 7f7a848f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:33.162 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.161+0000 7f7a7d59a640 1 -- 172.21.15.122:0/2640923 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a740017f0 con 0x7f7a8013fd70 2024-03-25T18:20:33.162 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.161+0000 7f7a7d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:33.162 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.161+0000 7f7a7d59a640 1 -- 172.21.15.122:0/2640923 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f7a74001c00 con 0x7f7a8013fd70 2024-03-25T18:20:33.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.161+0000 7f7a848f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:33.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.161+0000 7f7a848f6640 4 set_mon_vals no callback set 2024-03-25T18:20:33.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.161+0000 7f7a7d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:33.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.161+0000 7f7a7d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:33.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.161+0000 7f7a7d59a640 10 monclient: dump: 2024-03-25T18:20:33.163 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:33.163 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:33.163 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:33.163 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:33.163 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:33.163 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:33.163 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:33.163 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:33.163 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:33.163 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:33.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.162+0000 7f7a848f6640 10 monclient: shutdown 2024-03-25T18:20:33.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.162+0000 7f7a848f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:33.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.162+0000 7f7a848f6640 1 -- 172.21.15.122:0/2640923 >> v1:172.21.15.122:6789/0 conn(0x7f7a8013fd70 legacy=0x7f7a80145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.162+0000 7f7a848f6640 1 -- 172.21.15.122:0/2640923 shutdown_connections 2024-03-25T18:20:33.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.162+0000 7f7a848f6640 1 -- 172.21.15.122:0/2640923 >> 172.21.15.122:0/2640923 conn(0x7f7a800b4b80 msgr2=0x7f7a800b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:33.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.162+0000 7f7a848f6640 1 -- 172.21.15.122:0/2640923 shutdown_connections 2024-03-25T18:20:33.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.162+0000 7f7a848f6640 1 -- 172.21.15.122:0/2640923 wait complete. 2024-03-25T18:20:33.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.163+0000 7f7a848f6640 10 monclient: build_initial_monmap 2024-03-25T18:20:33.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.163+0000 7f7a848f6640 10 monclient: monmap: 2024-03-25T18:20:33.166 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:20:33.166 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:33.166 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:20:33.166 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:20:33.166 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:33.166 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:20:33.166 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:20:33.166 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:20:33.167 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:20:33.167 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:33.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.164+0000 7f7a848f6640 1 librados: starting msgr at 2024-03-25T18:20:33.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.164+0000 7f7a848f6640 1 librados: starting objecter 2024-03-25T18:20:33.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.164+0000 7f7a848f6640 1 Processor -- start 2024-03-25T18:20:33.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.164+0000 7f7a848f6640 1 -- start start 2024-03-25T18:20:33.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.164+0000 7f7a848f6640 1 librados: setting wanted keys 2024-03-25T18:20:33.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.164+0000 7f7a848f6640 1 librados: calling monclient init 2024-03-25T18:20:33.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.164+0000 7f7a848f6640 10 monclient: init 2024-03-25T18:20:33.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.165+0000 7f7a848f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:33.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.165+0000 7f7a848f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:33.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.165+0000 7f7a848f6640 10 monclient(hunting): picked mon.noname-c con 0x7f7a800afa60 addr v1:172.21.15.122:6790/0 2024-03-25T18:20:33.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.165+0000 7f7a848f6640 10 monclient(hunting): picked mon.noname-a con 0x7f7a8013f990 addr v1:172.21.15.122:6789/0 2024-03-25T18:20:33.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.165+0000 7f7a848f6640 10 monclient(hunting): picked mon.noname-b con 0x7f7a8013fd70 addr v1:172.21.15.204:6789/0 2024-03-25T18:20:33.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.165+0000 7f7a848f6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a8013cc80 con 0x7f7a8013f990 2024-03-25T18:20:33.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.165+0000 7f7a848f6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a801db680 con 0x7f7a8013fd70 2024-03-25T18:20:33.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.165+0000 7f7a848f6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a801db810 con 0x7f7a800afa60 2024-03-25T18:20:33.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.165+0000 7f7a848f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:33.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.165+0000 7f7a848f6640 10 monclient(hunting): authenticate will time out at 7556.431265s 2024-03-25T18:20:33.170 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.169+0000 7f7a7ed9d640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f7a8013fd70 0x7f7a8011dbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:43288/0 (socket says 172.21.15.122:43288) 2024-03-25T18:20:33.170 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.169+0000 7f7a7ed9d640 1 -- 172.21.15.122:0/4071269860 learned_addr learned my addr 172.21.15.122:0/4071269860 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:20:33.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.170+0000 7f7a6effd640 10 client.?.objecter ms_handle_connect 0x7f7a8013fd70 2024-03-25T18:20:33.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.170+0000 7f7a6effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:33.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.171+0000 7f7a6effd640 10 client.?.objecter ms_handle_connect 0x7f7a8013f990 2024-03-25T18:20:33.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.171+0000 7f7a6effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:33.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.178+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596173843 0 0) 0x7f7a801db680 con 0x7f7a8013fd70 2024-03-25T18:20:33.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.179+0000 7f7a6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:33.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.179+0000 7f7a6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:33.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.179+0000 7f7a6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:33.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.179+0000 7f7a6effd640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:20:33.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.179+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a60004eb0 con 0x7f7a8013fd70 2024-03-25T18:20:33.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.187+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870244759 0 0) 0x7f7a8013cc80 con 0x7f7a8013f990 2024-03-25T18:20:33.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.187+0000 7f7a6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:33.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.187+0000 7f7a6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:33.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.187+0000 7f7a6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:33.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.187+0000 7f7a6effd640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:20:33.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.187+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a801db680 con 0x7f7a8013f990 2024-03-25T18:20:33.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.188+0000 7f7a6effd640 10 client.?.objecter ms_handle_connect 0x7f7a800afa60 2024-03-25T18:20:33.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.188+0000 7f7a6effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:33.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.192+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3230966989 0 0) 0x7f7a60004eb0 con 0x7f7a8013fd70 2024-03-25T18:20:33.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.192+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a8013cc80 con 0x7f7a8013fd70 2024-03-25T18:20:33.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.192+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f7a78001cf0 con 0x7f7a8013fd70 2024-03-25T18:20:33.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.192+0000 7f7a6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:33.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.192+0000 7f7a6effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:33.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.192+0000 7f7a6effd640 10 monclient(hunting): dump: 2024-03-25T18:20:33.193 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:33.193 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:33.194 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:33.194 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:33.194 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:33.194 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:33.194 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:33.194 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:33.194 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:33.194 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:33.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.192+0000 7f7a6effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:33.197 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.195+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1113915910 0 0) 0x7f7a801db680 con 0x7f7a8013f990 2024-03-25T18:20:33.197 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.196+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7a60004eb0 con 0x7f7a8013f990 2024-03-25T18:20:33.197 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.196+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f7a74001b00 con 0x7f7a8013f990 2024-03-25T18:20:33.197 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.196+0000 7f7a6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:33.197 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.196+0000 7f7a6effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:33.197 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.196+0000 7f7a6effd640 10 monclient(hunting): dump: 2024-03-25T18:20:33.197 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:33.197 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:33.197 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:33.197 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:33.197 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:33.197 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:33.198 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:33.198 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:33.198 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:33.198 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:33.198 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.197+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337705328 0 0) 0x7f7a801db810 con 0x7f7a800afa60 2024-03-25T18:20:33.198 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.197+0000 7f7a6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:33.198 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.197+0000 7f7a6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:33.198 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.197+0000 7f7a6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:33.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.197+0000 7f7a6effd640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:20:33.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.197+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a801db680 con 0x7f7a800afa60 2024-03-25T18:20:33.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3151945647 0 0) 0x7f7a60004eb0 con 0x7f7a8013f990 2024-03-25T18:20:33.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 >> v1:172.21.15.122:6790/0 conn(0x7f7a800afa60 legacy=0x7f7a801daf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 >> v1:172.21.15.204:6789/0 conn(0x7f7a8013fd70 legacy=0x7f7a8011dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a6effd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:33.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a6effd640 1 monclient: found mon.a 2024-03-25T18:20:33.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:33.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a6effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:20:33.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8013c150 con 0x7f7a8013f990 2024-03-25T18:20:33.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a6effd640 10 monclient: _finish_auth 0 2024-03-25T18:20:33.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a6effd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:33.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:33.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a848f6640 5 monclient: authenticate success, global_id 4416 2024-03-25T18:20:33.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a848f6640 10 monclient: _renew_subs 2024-03-25T18:20:33.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a848f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:20:33.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a848f6640 1 -- 172.21.15.122:0/4071269860 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a8013c3a0 con 0x7f7a8013f990 2024-03-25T18:20:33.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a848f6640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:33.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a848f6640 10 monclient: _renew_subs 2024-03-25T18:20:33.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a848f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:20:33.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a848f6640 1 -- 172.21.15.122:0/4071269860 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a801e34a0 con 0x7f7a8013f990 2024-03-25T18:20:33.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.205+0000 7f7a848f6640 1 librados: init done 2024-03-25T18:20:33.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.206+0000 7f7a848f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:33.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.206+0000 7f7a848f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:33.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.206+0000 7f7a848f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:20:33.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.206+0000 7f7a848f6640 1 -- 172.21.15.122:0/4071269860 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a801dbba0 con 0x7f7a8013f990 2024-03-25T18:20:33.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.512+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7a74001ce0 con 0x7f7a8013f990 2024-03-25T18:20:33.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.512+0000 7f7a6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:33.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.512+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f7a74003a40 con 0x7f7a8013f990 2024-03-25T18:20:33.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.512+0000 7f7a6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:33.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.512+0000 7f7a6effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:33.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.512+0000 7f7a6effd640 10 monclient: dump: 2024-03-25T18:20:33.514 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:33.514 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:33.514 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:33.514 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:33.514 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:33.514 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:33.514 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:33.514 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:33.514 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:33.514 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:33.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.512+0000 7f7a7cd99640 4 set_mon_vals no callback set 2024-03-25T18:20:33.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.513+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f7a74019a10 con 0x7f7a8013f990 2024-03-25T18:20:33.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.513+0000 7f7a7e59c640 10 monclient: get_auth_request con 0x7f7a6006deb0 auth_method 0 2024-03-25T18:20:33.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.514+0000 7f7a6effd640 10 client.4416.objecter ms_handle_connect 0x7f7a6006deb0 2024-03-25T18:20:33.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.519+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f7a740535f0 con 0x7f7a8013f990 2024-03-25T18:20:33.521 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.519+0000 7f7a6effd640 10 client.4416.objecter ms_dispatch 0x7f7a80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:33.521 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.519+0000 7f7a6effd640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:33.521 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.519+0000 7f7a6effd640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:33.521 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.519+0000 7f7a6effd640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-25T18:20:33.521 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.519+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7a7404ef30 con 0x7f7a8013f990 2024-03-25T18:20:33.521 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.519+0000 7f7a6effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:33.521 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.519+0000 7f7a6effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:33.762 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.761+0000 7f7a848f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:33.762 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.761+0000 7f7a848f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:33.762 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.761+0000 7f7a848f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:20:33.762 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.761+0000 7f7a848f6640 1 -- 172.21.15.122:0/4071269860 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7a800627b0 con 0x7f7a8013f990 2024-03-25T18:20:33.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.786+0000 7f7a6effd640 1 -- 172.21.15.122:0/4071269860 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f7a74051030 con 0x7f7a8013f990 2024-03-25T18:20:33.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.786+0000 7f7a6effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:33.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.786+0000 7f7a6effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:33.788 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:20:33.788 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:33.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.788+0000 7f7a848f6640 10 librados: watch_flush enter 2024-03-25T18:20:33.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.789+0000 7f7a848f6640 10 librados: watch_flush exit 2024-03-25T18:20:33.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.789+0000 7f7a848f6640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-25T18:20:33.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.789+0000 7f7a848f6640 10 client.4416.objecter successfully canceled tick 2024-03-25T18:20:33.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.789+0000 7f7a848f6640 1 -- 172.21.15.122:0/4071269860 >> v1:172.21.15.204:6800/676228005 conn(0x7f7a6006deb0 legacy=0x7f7a60070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.789+0000 7f7a848f6640 10 monclient: shutdown 2024-03-25T18:20:33.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.789+0000 7f7a848f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:33.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.789+0000 7f7a848f6640 1 -- 172.21.15.122:0/4071269860 >> v1:172.21.15.122:6789/0 conn(0x7f7a8013f990 legacy=0x7f7a8011d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.789+0000 7f7a848f6640 1 -- 172.21.15.122:0/4071269860 shutdown_connections 2024-03-25T18:20:33.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.789+0000 7f7a848f6640 1 -- 172.21.15.122:0/4071269860 >> 172.21.15.122:0/4071269860 conn(0x7f7a800b4b80 msgr2=0x7f7a800b1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:33.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.789+0000 7f7a848f6640 1 -- 172.21.15.122:0/4071269860 shutdown_connections 2024-03-25T18:20:33.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.789+0000 7f7a848f6640 1 -- 172.21.15.122:0/4071269860 wait complete. 2024-03-25T18:20:33.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:33.789+0000 7f7a848f6640 1 librados: shutdown 2024-03-25T18:20:33.802 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:40.608 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:40.721 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.721+0000 7f4d03d5d640 10 monclient: get_monmap_and_config 2024-03-25T18:20:40.721 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.721+0000 7f4d03d5d640 10 monclient: build_initial_monmap 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.721+0000 7f4d03d5d640 10 monclient: monmap: 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:20:40.721853+0000 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:20:40.721853+0000 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:40.723 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.722+0000 7f4d03d5d640 1 Processor -- start 2024-03-25T18:20:40.723 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.722+0000 7f4d03d5d640 1 -- start start 2024-03-25T18:20:40.723 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.722+0000 7f4d03d5d640 10 monclient: init 2024-03-25T18:20:40.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.723+0000 7f4d03d5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:40.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.723+0000 7f4d03d5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:40.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.723+0000 7f4d03d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f4cfc0a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:20:40.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.723+0000 7f4d03d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f4cfc13f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:20:40.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.723+0000 7f4d03d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f4cfc1430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:20:40.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.723+0000 7f4d03d5d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc142520 con 0x7f4cfc1430c0 2024-03-25T18:20:40.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.723+0000 7f4d03d5d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc0b28a0 con 0x7f4cfc0a98f0 2024-03-25T18:20:40.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.723+0000 7f4d03d5d640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc0b2a30 con 0x7f4cfc13f710 2024-03-25T18:20:40.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.723+0000 7f4d03d5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:40.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.723+0000 7f4d03d5d640 10 monclient(hunting): authenticate will time out at 7563.988171s 2024-03-25T18:20:40.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.723+0000 7f4d012f8640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f4cfc13f710 0x7f4cfc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:60084/0 (socket says 172.21.15.122:60084) 2024-03-25T18:20:40.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.723+0000 7f4d012f8640 1 -- 172.21.15.122:0/1936711922 learned_addr learned my addr 172.21.15.122:0/1936711922 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:20:40.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.783+0000 7f4d00af7640 1 -- 172.21.15.122:0/1936711922 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3792163797 0 0) 0x7f4cfc142520 con 0x7f4cfc1430c0 2024-03-25T18:20:40.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.783+0000 7f4d00af7640 10 monclient(hunting): _negotiate 2024-03-25T18:20:40.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.783+0000 7f4d00af7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:40.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.783+0000 7f4d00af7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:40.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.783+0000 7f4d00af7640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:20:40.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.783+0000 7f4d00af7640 1 -- 172.21.15.122:0/1936711922 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ce8003520 con 0x7f4cfc1430c0 2024-03-25T18:20:40.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d00af7640 1 -- 172.21.15.122:0/1936711922 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1031797993 0 0) 0x7f4ce8003520 con 0x7f4cfc1430c0 2024-03-25T18:20:40.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d00af7640 1 -- 172.21.15.122:0/1936711922 >> v1:172.21.15.122:6790/0 conn(0x7f4cfc13f710 legacy=0x7f4cfc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d00af7640 1 -- 172.21.15.122:0/1936711922 >> v1:172.21.15.204:6789/0 conn(0x7f4cfc0a98f0 legacy=0x7f4cfc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d00af7640 10 monclient: _finish_hunting 0 2024-03-25T18:20:40.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d00af7640 1 monclient: found mon.noname-a 2024-03-25T18:20:40.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d00af7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:40.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d00af7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:20:40.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d00af7640 1 -- 172.21.15.122:0/1936711922 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cfc0b44c0 con 0x7f4cfc1430c0 2024-03-25T18:20:40.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d00af7640 10 monclient: _finish_auth 0 2024-03-25T18:20:40.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d00af7640 10 monclient: _check_auth_tickets 2024-03-25T18:20:40.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d00af7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:40.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d00af7640 1 -- 172.21.15.122:0/1936711922 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4cf4001a10 con 0x7f4cfc1430c0 2024-03-25T18:20:40.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d03d5d640 5 monclient: authenticate success, global_id 4419 2024-03-25T18:20:40.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d03d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:40.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d00af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d00af7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:40.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d00af7640 10 monclient: dump: 2024-03-25T18:20:40.796 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:40.796 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:40.796 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:40.797 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:40.797 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.797 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:40.797 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:40.797 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:40.797 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:40.797 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:40.797 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.795+0000 7f4d03d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:40.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.804+0000 7f4d00af7640 1 -- 172.21.15.122:0/1936711922 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cf4001b70 con 0x7f4cfc1430c0 2024-03-25T18:20:40.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.804+0000 7f4d00af7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:40.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.804+0000 7f4d00af7640 1 -- 172.21.15.122:0/1936711922 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4cf4001f80 con 0x7f4cfc1430c0 2024-03-25T18:20:40.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.804+0000 7f4d03d5d640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:40.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.804+0000 7f4d03d5d640 4 set_mon_vals no callback set 2024-03-25T18:20:40.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.804+0000 7f4d00af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.804+0000 7f4d00af7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:40.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.804+0000 7f4d00af7640 10 monclient: dump: 2024-03-25T18:20:40.804 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:40.804 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:40.805 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:40.805 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:40.805 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.805 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:40.805 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:40.805 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:40.805 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:40.805 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:40.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.805+0000 7f4d03d5d640 10 monclient: shutdown 2024-03-25T18:20:40.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.805+0000 7f4d03d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:40.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.805+0000 7f4d03d5d640 1 -- 172.21.15.122:0/1936711922 >> v1:172.21.15.122:6789/0 conn(0x7f4cfc1430c0 legacy=0x7f4cfc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.805+0000 7f4d03d5d640 1 -- 172.21.15.122:0/1936711922 shutdown_connections 2024-03-25T18:20:40.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.805+0000 7f4d03d5d640 1 -- 172.21.15.122:0/1936711922 >> 172.21.15.122:0/1936711922 conn(0x7f4cfc0b7ea0 msgr2=0x7f4cfc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:40.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.805+0000 7f4d03d5d640 1 -- 172.21.15.122:0/1936711922 shutdown_connections 2024-03-25T18:20:40.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.805+0000 7f4d03d5d640 1 -- 172.21.15.122:0/1936711922 wait complete. 2024-03-25T18:20:40.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.806+0000 7f4d03d5d640 10 monclient: build_initial_monmap 2024-03-25T18:20:40.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.806+0000 7f4d03d5d640 10 monclient: monmap: 2024-03-25T18:20:40.806 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:20:40.807 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:40.807 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:20:40.807 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:20:40.807 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:40.807 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:20:40.807 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:20:40.807 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:20:40.807 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:20:40.807 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:40.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.806+0000 7f4d03d5d640 1 librados: starting msgr at 2024-03-25T18:20:40.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.806+0000 7f4d03d5d640 1 librados: starting objecter 2024-03-25T18:20:40.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.806+0000 7f4d03d5d640 1 Processor -- start 2024-03-25T18:20:40.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.806+0000 7f4d03d5d640 1 -- start start 2024-03-25T18:20:40.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.806+0000 7f4d03d5d640 1 librados: setting wanted keys 2024-03-25T18:20:40.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.806+0000 7f4d03d5d640 1 librados: calling monclient init 2024-03-25T18:20:40.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.806+0000 7f4d03d5d640 10 monclient: init 2024-03-25T18:20:40.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.807+0000 7f4d03d5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:40.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.807+0000 7f4d03d5d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:40.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.807+0000 7f4d03d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f4cfc0a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:20:40.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.807+0000 7f4d03d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f4cfc13f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:20:40.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.807+0000 7f4d03d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f4cfc1430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:20:40.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.807+0000 7f4d03d5d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc13b9e0 con 0x7f4cfc1430c0 2024-03-25T18:20:40.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.807+0000 7f4d03d5d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc14dd30 con 0x7f4cfc13f710 2024-03-25T18:20:40.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.807+0000 7f4d03d5d640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cfc135680 con 0x7f4cfc0a98f0 2024-03-25T18:20:40.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.807+0000 7f4d03d5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:40.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.807+0000 7f4d03d5d640 10 monclient(hunting): authenticate will time out at 7564.072588s 2024-03-25T18:20:40.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.817+0000 7f4d022fa640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4cfc1430c0 0x7f4cfc134f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37006/0 (socket says 172.21.15.122:37006) 2024-03-25T18:20:40.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.817+0000 7f4d022fa640 1 -- 172.21.15.122:0/2145127588 learned_addr learned my addr 172.21.15.122:0/2145127588 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:20:40.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.819+0000 7f4cf27fc640 10 client.?.objecter ms_handle_connect 0x7f4cfc1430c0 2024-03-25T18:20:40.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.819+0000 7f4cf27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.833+0000 7f4cf27fc640 1 -- 172.21.15.122:0/2145127588 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3194483785 0 0) 0x7f4cfc13b9e0 con 0x7f4cfc1430c0 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.833+0000 7f4cf27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.833+0000 7f4cf27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.833+0000 7f4cf27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:40.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.833+0000 7f4cf27fc640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:20:40.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.833+0000 7f4cf27fc640 1 -- 172.21.15.122:0/2145127588 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ce0004eb0 con 0x7f4cfc1430c0 2024-03-25T18:20:40.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.834+0000 7f4cf27fc640 10 client.?.objecter ms_handle_connect 0x7f4cfc13f710 2024-03-25T18:20:40.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.834+0000 7f4cf27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:40.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.849+0000 7f4cf27fc640 1 -- 172.21.15.122:0/2145127588 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1399889587 0 0) 0x7f4ce0004eb0 con 0x7f4cfc1430c0 2024-03-25T18:20:40.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.849+0000 7f4cf27fc640 1 -- 172.21.15.122:0/2145127588 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4cfc13b9e0 con 0x7f4cfc1430c0 2024-03-25T18:20:40.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.849+0000 7f4cf27fc640 1 -- 172.21.15.122:0/2145127588 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4cf4001fc0 con 0x7f4cfc1430c0 2024-03-25T18:20:40.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.849+0000 7f4cf27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.849+0000 7f4cf27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:40.850 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.849+0000 7f4cf27fc640 10 monclient(hunting): dump: 2024-03-25T18:20:40.850 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:40.850 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:40.850 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:40.850 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:40.850 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.850 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:40.850 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:40.850 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:40.850 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:40.850 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:40.850 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.849+0000 7f4cf27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:40.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.852+0000 7f4cf27fc640 10 client.?.objecter ms_handle_connect 0x7f4cfc0a98f0 2024-03-25T18:20:40.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.852+0000 7f4cf27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:40.857 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4cf27fc640 1 -- 172.21.15.122:0/2145127588 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3242287669 0 0) 0x7f4cfc13b9e0 con 0x7f4cfc1430c0 2024-03-25T18:20:40.857 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4cf27fc640 1 -- 172.21.15.122:0/2145127588 >> v1:172.21.15.122:6790/0 conn(0x7f4cfc0a98f0 legacy=0x7f4cfc14e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.857 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4cf27fc640 1 -- 172.21.15.122:0/2145127588 >> v1:172.21.15.204:6789/0 conn(0x7f4cfc13f710 legacy=0x7f4cfc134860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4cf27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4cf27fc640 1 monclient: found mon.a 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4cf27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4cf27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4cf27fc640 1 -- 172.21.15.122:0/2145127588 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cfc13be70 con 0x7f4cfc1430c0 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4cf27fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4cf27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4cf27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4d03d5d640 5 monclient: authenticate success, global_id 4422 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4d03d5d640 10 monclient: _renew_subs 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4d03d5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4d03d5d640 1 -- 172.21.15.122:0/2145127588 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cfc14caa0 con 0x7f4cfc1430c0 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4d03d5d640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4d03d5d640 10 monclient: _renew_subs 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4d03d5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:20:40.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4d03d5d640 1 -- 172.21.15.122:0/2145127588 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cfc14c730 con 0x7f4cfc1430c0 2024-03-25T18:20:40.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.857+0000 7f4d03d5d640 1 librados: init done 2024-03-25T18:20:40.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.858+0000 7f4d03d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:40.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.858+0000 7f4d03d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:40.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.858+0000 7f4d03d5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:20:40.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:40.858+0000 7f4d03d5d640 1 -- 172.21.15.122:0/2145127588 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cfc0627b0 con 0x7f4cfc1430c0 2024-03-25T18:20:41.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.102+0000 7f4cf27fc640 1 -- 172.21.15.122:0/2145127588 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4cf4003c20 con 0x7f4cfc1430c0 2024-03-25T18:20:41.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.102+0000 7f4cf27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:41.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.102+0000 7f4cf27fc640 1 -- 172.21.15.122:0/2145127588 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4cf4003ff0 con 0x7f4cfc1430c0 2024-03-25T18:20:41.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.102+0000 7f4cf27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:41.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.102+0000 7f4cf27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:41.103 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.102+0000 7f4cf27fc640 10 monclient: dump: 2024-03-25T18:20:41.103 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:41.103 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:41.103 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:41.103 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:41.103 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:41.103 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:41.103 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:41.104 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:41.104 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:41.104 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:41.104 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.102+0000 7f4cf3fff640 4 set_mon_vals no callback set 2024-03-25T18:20:41.104 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.103+0000 7f4cf27fc640 1 -- 172.21.15.122:0/2145127588 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f4cf4004210 con 0x7f4cfc1430c0 2024-03-25T18:20:41.104 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.104+0000 7f4d01af9640 10 monclient: get_auth_request con 0x7f4ce006dd00 auth_method 0 2024-03-25T18:20:41.104 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.104+0000 7f4cf27fc640 10 client.4422.objecter ms_handle_connect 0x7f4ce006dd00 2024-03-25T18:20:41.108 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.108+0000 7f4cf27fc640 1 -- 172.21.15.122:0/2145127588 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f4cf4054c80 con 0x7f4cfc1430c0 2024-03-25T18:20:41.108 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.108+0000 7f4cf27fc640 10 client.4422.objecter ms_dispatch 0x7f4cfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:41.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.108+0000 7f4cf27fc640 3 client.4422.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:41.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.108+0000 7f4cf27fc640 3 client.4422.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:41.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.108+0000 7f4cf27fc640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-25T18:20:41.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.108+0000 7f4cf27fc640 1 -- 172.21.15.122:0/2145127588 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4cf4050790 con 0x7f4cfc1430c0 2024-03-25T18:20:41.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.108+0000 7f4cf27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:41.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.108+0000 7f4cf27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:41.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.339+0000 7f4d03d5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:41.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.339+0000 7f4d03d5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:41.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.339+0000 7f4d03d5d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:20:41.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.339+0000 7f4d03d5d640 1 -- 172.21.15.122:0/2145127588 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4cfc13df50 con 0x7f4cfc1430c0 2024-03-25T18:20:41.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.363+0000 7f4cf27fc640 1 -- 172.21.15.122:0/2145127588 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f4cf40526c0 con 0x7f4cfc1430c0 2024-03-25T18:20:41.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.363+0000 7f4cf27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:41.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.363+0000 7f4cf27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:41.363 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:20:41.363 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:41.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.365+0000 7f4d03d5d640 10 librados: watch_flush enter 2024-03-25T18:20:41.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.365+0000 7f4d03d5d640 10 librados: watch_flush exit 2024-03-25T18:20:41.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.365+0000 7f4d03d5d640 20 client.4422.objecter shutdown clearing up homeless session... 2024-03-25T18:20:41.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.365+0000 7f4d03d5d640 10 client.4422.objecter successfully canceled tick 2024-03-25T18:20:41.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.365+0000 7f4d03d5d640 1 -- 172.21.15.122:0/2145127588 >> v1:172.21.15.204:6800/676228005 conn(0x7f4ce006dd00 legacy=0x7f4ce00701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:41.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.365+0000 7f4d03d5d640 10 monclient: shutdown 2024-03-25T18:20:41.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.365+0000 7f4d03d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:41.367 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.365+0000 7f4d03d5d640 1 -- 172.21.15.122:0/2145127588 >> v1:172.21.15.122:6789/0 conn(0x7f4cfc1430c0 legacy=0x7f4cfc134f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:41.367 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.365+0000 7f4d03d5d640 1 -- 172.21.15.122:0/2145127588 shutdown_connections 2024-03-25T18:20:41.367 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.365+0000 7f4d03d5d640 1 -- 172.21.15.122:0/2145127588 >> 172.21.15.122:0/2145127588 conn(0x7f4cfc0b7ea0 msgr2=0x7f4cfc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:41.367 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.365+0000 7f4d03d5d640 1 -- 172.21.15.122:0/2145127588 shutdown_connections 2024-03-25T18:20:41.367 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.366+0000 7f4d03d5d640 1 -- 172.21.15.122:0/2145127588 wait complete. 2024-03-25T18:20:41.367 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:41.366+0000 7f4d03d5d640 1 librados: shutdown 2024-03-25T18:20:41.377 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:46.222 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:04:39.964 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:46.229 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:04:39.972 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:46.276 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:04:40.018 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:46.276 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:04:40.022 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:46.289 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:04:40.034 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:46.294 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:04:40.036 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:46.303 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:04:40.046 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:46.317 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:04:40.062 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:46.349 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:04:40.092 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:48.184 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:48.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.300+0000 7fc9df055640 10 monclient: get_monmap_and_config 2024-03-25T18:20:48.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.300+0000 7fc9df055640 10 monclient: build_initial_monmap 2024-03-25T18:20:48.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.300+0000 7fc9df055640 10 monclient: monmap: 2024-03-25T18:20:48.302 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:20:48.302 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:48.302 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:20:48.302039+0000 2024-03-25T18:20:48.302 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:20:48.302039+0000 2024-03-25T18:20:48.302 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:48.302 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:48.303 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:20:48.303 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:20:48.303 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:20:48.303 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:48.304 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.302+0000 7fc9df055640 1 Processor -- start 2024-03-25T18:20:48.304 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.302+0000 7fc9df055640 1 -- start start 2024-03-25T18:20:48.304 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.302+0000 7fc9df055640 10 monclient: init 2024-03-25T18:20:48.304 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.302+0000 7fc9df055640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:48.304 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.302+0000 7fc9df055640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:48.304 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.302+0000 7fc9df055640 10 monclient(hunting): picked mon.noname-c con 0x7fc9d80a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:20:48.304 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.302+0000 7fc9df055640 10 monclient(hunting): picked mon.noname-a con 0x7fc9d813f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:20:48.304 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.303+0000 7fc9df055640 10 monclient(hunting): picked mon.noname-b con 0x7fc9d81430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:20:48.305 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.303+0000 7fc9df055640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d8142520 con 0x7fc9d813f710 2024-03-25T18:20:48.305 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.303+0000 7fc9df055640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d80b28a0 con 0x7fc9d80a98f0 2024-03-25T18:20:48.305 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.303+0000 7fc9df055640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d80b2a30 con 0x7fc9d81430c0 2024-03-25T18:20:48.305 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.303+0000 7fc9df055640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:48.305 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.303+0000 7fc9df055640 10 monclient(hunting): authenticate will time out at 7571.569161s 2024-03-25T18:20:48.311 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.310+0000 7fc9dcdf1640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fc9d80a98f0 0x7fc9d80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:42748/0 (socket says 172.21.15.122:42748) 2024-03-25T18:20:48.311 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.310+0000 7fc9dcdf1640 1 -- 172.21.15.122:0/2550140837 learned_addr learned my addr 172.21.15.122:0/2550140837 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:20:48.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.320+0000 7fc9d77fe640 1 -- 172.21.15.122:0/2550140837 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738544656 0 0) 0x7fc9d80b2a30 con 0x7fc9d81430c0 2024-03-25T18:20:48.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.320+0000 7fc9d77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:48.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.320+0000 7fc9d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:48.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.320+0000 7fc9d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:48.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.320+0000 7fc9d77fe640 10 monclient(hunting): my global_id is 4397 2024-03-25T18:20:48.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.320+0000 7fc9d77fe640 1 -- 172.21.15.122:0/2550140837 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9c8003520 con 0x7fc9d81430c0 2024-03-25T18:20:48.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.326+0000 7fc9d77fe640 1 -- 172.21.15.122:0/2550140837 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893228410 0 0) 0x7fc9d8142520 con 0x7fc9d813f710 2024-03-25T18:20:48.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.326+0000 7fc9d77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:48.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.326+0000 7fc9d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:48.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.326+0000 7fc9d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:48.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.326+0000 7fc9d77fe640 10 monclient(hunting): my global_id is 4425 2024-03-25T18:20:48.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.326+0000 7fc9d77fe640 1 -- 172.21.15.122:0/2550140837 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9d80b2a30 con 0x7fc9d813f710 2024-03-25T18:20:48.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.328+0000 7fc9d77fe640 1 -- 172.21.15.122:0/2550140837 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 35422721 0 0) 0x7fc9d80b28a0 con 0x7fc9d80a98f0 2024-03-25T18:20:48.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.328+0000 7fc9d77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:48.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.328+0000 7fc9d77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:48.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.328+0000 7fc9d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:48.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.328+0000 7fc9d77fe640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:20:48.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.328+0000 7fc9d77fe640 1 -- 172.21.15.122:0/2550140837 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9d8142520 con 0x7fc9d80a98f0 2024-03-25T18:20:48.337 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.336+0000 7fc9d77fe640 1 -- 172.21.15.122:0/2550140837 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1565889936 0 0) 0x7fc9d8142520 con 0x7fc9d80a98f0 2024-03-25T18:20:48.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.336+0000 7fc9d77fe640 1 -- 172.21.15.122:0/2550140837 >> v1:172.21.15.122:6790/0 conn(0x7fc9d81430c0 legacy=0x7fc9d814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.336+0000 7fc9d77fe640 1 -- 172.21.15.122:0/2550140837 >> v1:172.21.15.122:6789/0 conn(0x7fc9d813f710 legacy=0x7fc9d81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.336+0000 7fc9d77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:48.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.336+0000 7fc9d77fe640 1 monclient: found mon.noname-c 2024-03-25T18:20:48.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.336+0000 7fc9d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:48.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.336+0000 7fc9d77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:20:48.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.336+0000 7fc9d77fe640 1 -- 172.21.15.122:0/2550140837 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d80b44c0 con 0x7fc9d80a98f0 2024-03-25T18:20:48.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.336+0000 7fc9d77fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:48.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.336+0000 7fc9d77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:48.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.336+0000 7fc9d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:48.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.336+0000 7fc9d77fe640 1 -- 172.21.15.122:0/2550140837 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fc9cc0015b0 con 0x7fc9d80a98f0 2024-03-25T18:20:48.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.336+0000 7fc9d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:48.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.336+0000 7fc9d77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:48.339 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.336+0000 7fc9d77fe640 10 monclient: dump: 2024-03-25T18:20:48.339 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:48.339 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:48.339 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:48.339 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:48.339 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:48.339 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:48.339 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:48.339 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:48.339 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:48.339 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:48.339 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.336+0000 7fc9df055640 5 monclient: authenticate success, global_id 4420 2024-03-25T18:20:48.339 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.336+0000 7fc9df055640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:48.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.354+0000 7fc9d77fe640 1 -- 172.21.15.122:0/2550140837 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9cc001790 con 0x7fc9d80a98f0 2024-03-25T18:20:48.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.354+0000 7fc9d77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:48.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.354+0000 7fc9d77fe640 1 -- 172.21.15.122:0/2550140837 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fc9cc001be0 con 0x7fc9d80a98f0 2024-03-25T18:20:48.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.354+0000 7fc9d77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:48.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.354+0000 7fc9d77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:48.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.354+0000 7fc9d77fe640 10 monclient: dump: 2024-03-25T18:20:48.355 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:48.355 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:48.355 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:48.355 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:48.355 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:48.356 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:48.356 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:48.356 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:48.356 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:48.356 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:48.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.354+0000 7fc9df055640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:48.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.354+0000 7fc9df055640 4 set_mon_vals no callback set 2024-03-25T18:20:48.357 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.354+0000 7fc9df055640 10 monclient: shutdown 2024-03-25T18:20:48.357 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.354+0000 7fc9df055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:48.357 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.354+0000 7fc9df055640 1 -- 172.21.15.122:0/2550140837 >> v1:172.21.15.204:6789/0 conn(0x7fc9d80a98f0 legacy=0x7fc9d80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.357 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.354+0000 7fc9df055640 1 -- 172.21.15.122:0/2550140837 shutdown_connections 2024-03-25T18:20:48.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.354+0000 7fc9df055640 1 -- 172.21.15.122:0/2550140837 >> 172.21.15.122:0/2550140837 conn(0x7fc9d80b7ea0 msgr2=0x7fc9d80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:48.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.355+0000 7fc9df055640 1 -- 172.21.15.122:0/2550140837 shutdown_connections 2024-03-25T18:20:48.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.355+0000 7fc9df055640 1 -- 172.21.15.122:0/2550140837 wait complete. 2024-03-25T18:20:48.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.355+0000 7fc9df055640 10 monclient: build_initial_monmap 2024-03-25T18:20:48.358 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.355+0000 7fc9df055640 10 monclient: monmap: 2024-03-25T18:20:48.358 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:20:48.358 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:48.358 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:20:48.358 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:20:48.358 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:48.358 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:20:48.358 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:20:48.358 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:20:48.358 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:20:48.359 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:48.359 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.356+0000 7fc9df055640 1 librados: starting msgr at 2024-03-25T18:20:48.359 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.356+0000 7fc9df055640 1 librados: starting objecter 2024-03-25T18:20:48.359 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.356+0000 7fc9df055640 1 Processor -- start 2024-03-25T18:20:48.359 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.356+0000 7fc9df055640 1 -- start start 2024-03-25T18:20:48.359 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.356+0000 7fc9df055640 1 librados: setting wanted keys 2024-03-25T18:20:48.359 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.356+0000 7fc9df055640 1 librados: calling monclient init 2024-03-25T18:20:48.359 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.356+0000 7fc9df055640 10 monclient: init 2024-03-25T18:20:48.360 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.357+0000 7fc9df055640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:48.360 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.357+0000 7fc9df055640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:48.360 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.357+0000 7fc9df055640 10 monclient(hunting): picked mon.noname-c con 0x7fc9d80a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:20:48.360 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.357+0000 7fc9df055640 10 monclient(hunting): picked mon.noname-b con 0x7fc9d813f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:20:48.360 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.357+0000 7fc9df055640 10 monclient(hunting): picked mon.noname-a con 0x7fc9d81430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:20:48.360 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.357+0000 7fc9df055640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d81db410 con 0x7fc9d81430c0 2024-03-25T18:20:48.360 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.357+0000 7fc9df055640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d81399a0 con 0x7fc9d813f710 2024-03-25T18:20:48.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.357+0000 7fc9df055640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9d80b08d0 con 0x7fc9d80a98f0 2024-03-25T18:20:48.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.357+0000 7fc9df055640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:48.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.357+0000 7fc9df055640 10 monclient(hunting): authenticate will time out at 7571.623738s 2024-03-25T18:20:48.373 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.372+0000 7fc9dd5f2640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fc9d81430c0 0x7fc9d81d4270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:41456/0 (socket says 172.21.15.122:41456) 2024-03-25T18:20:48.373 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.372+0000 7fc9dd5f2640 1 -- 172.21.15.122:0/1110851954 learned_addr learned my addr 172.21.15.122:0/1110851954 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:20:48.375 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.374+0000 7fc9d57fa640 10 client.?.objecter ms_handle_connect 0x7fc9d81430c0 2024-03-25T18:20:48.375 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.374+0000 7fc9d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:48.381 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.380+0000 7fc9d57fa640 10 client.?.objecter ms_handle_connect 0x7fc9d813f710 2024-03-25T18:20:48.381 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.380+0000 7fc9d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:48.382 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.381+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545251389 0 0) 0x7fc9d81db410 con 0x7fc9d81430c0 2024-03-25T18:20:48.382 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.381+0000 7fc9d57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:48.382 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.381+0000 7fc9d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:48.382 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.381+0000 7fc9d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:48.383 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.381+0000 7fc9d57fa640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:20:48.383 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.381+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9bc004eb0 con 0x7fc9d81430c0 2024-03-25T18:20:48.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.384+0000 7fc9d57fa640 10 client.?.objecter ms_handle_connect 0x7fc9d80a98f0 2024-03-25T18:20:48.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.384+0000 7fc9d57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:48.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.385+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213714126 0 0) 0x7fc9d81399a0 con 0x7fc9d813f710 2024-03-25T18:20:48.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.385+0000 7fc9d57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:48.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.385+0000 7fc9d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:48.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.385+0000 7fc9d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:48.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.385+0000 7fc9d57fa640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:20:48.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.385+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9d81db410 con 0x7fc9d813f710 2024-03-25T18:20:48.392 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.391+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 121272649 0 0) 0x7fc9bc004eb0 con 0x7fc9d81430c0 2024-03-25T18:20:48.392 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.391+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9d81399a0 con 0x7fc9d81430c0 2024-03-25T18:20:48.392 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.391+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fc9d0001c30 con 0x7fc9d81430c0 2024-03-25T18:20:48.392 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.391+0000 7fc9d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:48.392 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.391+0000 7fc9d57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:48.393 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.391+0000 7fc9d57fa640 10 monclient(hunting): dump: 2024-03-25T18:20:48.393 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:48.393 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:48.393 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:48.393 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:48.393 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:48.393 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:48.393 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:48.393 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:48.393 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:48.393 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:48.393 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.391+0000 7fc9d57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:48.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.394+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 11389925 0 0) 0x7fc9d81db410 con 0x7fc9d813f710 2024-03-25T18:20:48.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.394+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9bc004eb0 con 0x7fc9d813f710 2024-03-25T18:20:48.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.394+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fc9c4001930 con 0x7fc9d813f710 2024-03-25T18:20:48.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.394+0000 7fc9d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:48.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.394+0000 7fc9d57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:48.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.394+0000 7fc9d57fa640 10 monclient(hunting): dump: 2024-03-25T18:20:48.396 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:48.396 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:48.396 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:48.396 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:48.396 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:48.396 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:48.396 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:48.396 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:48.396 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:48.396 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:48.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.398+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268949587 0 0) 0x7fc9d80b08d0 con 0x7fc9d80a98f0 2024-03-25T18:20:48.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.398+0000 7fc9d57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:48.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.398+0000 7fc9d57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:48.400 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.398+0000 7fc9d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:48.400 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.398+0000 7fc9d57fa640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:20:48.400 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.398+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9d81db410 con 0x7fc9d80a98f0 2024-03-25T18:20:48.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4026669242 0 0) 0x7fc9d81399a0 con 0x7fc9d81430c0 2024-03-25T18:20:48.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 >> v1:172.21.15.122:6790/0 conn(0x7fc9d80a98f0 legacy=0x7fc9d81dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 >> v1:172.21.15.204:6789/0 conn(0x7fc9d813f710 legacy=0x7fc9d80b01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9d57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:20:48.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9d57fa640 1 monclient: found mon.a 2024-03-25T18:20:48.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9d57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:48.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9d57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:20:48.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d80b0a60 con 0x7fc9d81430c0 2024-03-25T18:20:48.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9d57fa640 10 monclient: _finish_auth 0 2024-03-25T18:20:48.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9d57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:20:48.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:48.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9df055640 5 monclient: authenticate success, global_id 4428 2024-03-25T18:20:48.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9df055640 10 monclient: _renew_subs 2024-03-25T18:20:48.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9df055640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:20:48.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9df055640 1 -- 172.21.15.122:0/1110851954 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9d80b0e40 con 0x7fc9d81430c0 2024-03-25T18:20:48.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9df055640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:48.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9df055640 10 monclient: _renew_subs 2024-03-25T18:20:48.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9df055640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:20:48.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9df055640 1 -- 172.21.15.122:0/1110851954 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9d81d4b60 con 0x7fc9d81430c0 2024-03-25T18:20:48.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.404+0000 7fc9df055640 1 librados: init done 2024-03-25T18:20:48.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.405+0000 7fc9df055640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:48.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.405+0000 7fc9df055640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:48.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.405+0000 7fc9df055640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:20:48.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.405+0000 7fc9df055640 1 -- 172.21.15.122:0/1110851954 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9d80b11e0 con 0x7fc9d81430c0 2024-03-25T18:20:48.722 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.721+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9d00020e0 con 0x7fc9d81430c0 2024-03-25T18:20:48.722 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.721+0000 7fc9d57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:48.722 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.721+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fc9d0003b30 con 0x7fc9d81430c0 2024-03-25T18:20:48.722 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.721+0000 7fc9d57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:48.722 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.721+0000 7fc9d57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:48.722 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.721+0000 7fc9d57fa640 10 monclient: dump: 2024-03-25T18:20:48.723 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:48.723 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:48.723 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:48.723 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:48.723 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:48.723 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:48.723 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:48.723 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:48.723 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:48.723 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:48.723 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.721+0000 7fc9d6ffd640 4 set_mon_vals no callback set 2024-03-25T18:20:48.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.722+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fc9d0003d50 con 0x7fc9d81430c0 2024-03-25T18:20:48.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.722+0000 7fc9dcdf1640 10 monclient: get_auth_request con 0x7fc9bc06df30 auth_method 0 2024-03-25T18:20:48.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.723+0000 7fc9d57fa640 10 client.4428.objecter ms_handle_connect 0x7fc9bc06df30 2024-03-25T18:20:48.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.732+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fc9d0054a60 con 0x7fc9d81430c0 2024-03-25T18:20:48.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.732+0000 7fc9d57fa640 10 client.4428.objecter ms_dispatch 0x7fc9d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:48.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.732+0000 7fc9d57fa640 3 client.4428.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:48.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.732+0000 7fc9d57fa640 3 client.4428.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:48.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.732+0000 7fc9d57fa640 20 client.4428.objecter dump_active .. 0 homeless 2024-03-25T18:20:48.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.732+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9d0050570 con 0x7fc9d81430c0 2024-03-25T18:20:48.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.733+0000 7fc9d57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:48.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.733+0000 7fc9d57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:48.957 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.956+0000 7fc9df055640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:48.958 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.956+0000 7fc9df055640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:48.958 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.956+0000 7fc9df055640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:20:48.958 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.956+0000 7fc9df055640 1 -- 172.21.15.122:0/1110851954 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9d80b0fd0 con 0x7fc9d81430c0 2024-03-25T18:20:48.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.983+0000 7fc9d57fa640 1 -- 172.21.15.122:0/1110851954 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fc9d00524a0 con 0x7fc9d81430c0 2024-03-25T18:20:48.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.983+0000 7fc9d57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:48.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.983+0000 7fc9d57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:48.984 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:20:48.984 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:48.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.985+0000 7fc9df055640 10 librados: watch_flush enter 2024-03-25T18:20:48.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.985+0000 7fc9df055640 10 librados: watch_flush exit 2024-03-25T18:20:48.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.985+0000 7fc9df055640 20 client.4428.objecter shutdown clearing up homeless session... 2024-03-25T18:20:48.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.985+0000 7fc9df055640 10 client.4428.objecter successfully canceled tick 2024-03-25T18:20:48.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.985+0000 7fc9df055640 1 -- 172.21.15.122:0/1110851954 >> v1:172.21.15.204:6800/676228005 conn(0x7fc9bc06df30 legacy=0x7fc9bc0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.985+0000 7fc9df055640 10 monclient: shutdown 2024-03-25T18:20:48.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.985+0000 7fc9df055640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:48.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.985+0000 7fc9df055640 1 -- 172.21.15.122:0/1110851954 >> v1:172.21.15.122:6789/0 conn(0x7fc9d81430c0 legacy=0x7fc9d81d4270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.985+0000 7fc9df055640 1 -- 172.21.15.122:0/1110851954 shutdown_connections 2024-03-25T18:20:48.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.985+0000 7fc9df055640 1 -- 172.21.15.122:0/1110851954 >> 172.21.15.122:0/1110851954 conn(0x7fc9d80b7ea0 msgr2=0x7fc9d8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:48.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.985+0000 7fc9df055640 1 -- 172.21.15.122:0/1110851954 shutdown_connections 2024-03-25T18:20:48.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.985+0000 7fc9df055640 1 -- 172.21.15.122:0/1110851954 wait complete. 2024-03-25T18:20:48.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:48.985+0000 7fc9df055640 1 librados: shutdown 2024-03-25T18:20:48.996 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:55.803 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:55.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.915+0000 7f59beaf4640 10 monclient: get_monmap_and_config 2024-03-25T18:20:55.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.915+0000 7f59beaf4640 10 monclient: build_initial_monmap 2024-03-25T18:20:55.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.915+0000 7f59beaf4640 10 monclient: monmap: 2024-03-25T18:20:55.916 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:20:55.916 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:55.917 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:20:55.916572+0000 2024-03-25T18:20:55.917 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:20:55.916572+0000 2024-03-25T18:20:55.917 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:55.917 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:55.917 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:20:55.917 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:20:55.917 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:20:55.917 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:55.918 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.916+0000 7f59beaf4640 1 Processor -- start 2024-03-25T18:20:55.918 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.916+0000 7f59beaf4640 1 -- start start 2024-03-25T18:20:55.918 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.916+0000 7f59beaf4640 10 monclient: init 2024-03-25T18:20:55.919 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.916+0000 7f59beaf4640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:55.919 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.916+0000 7f59beaf4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:55.919 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.916+0000 7f59beaf4640 10 monclient(hunting): picked mon.noname-a con 0x7f59b80a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:20:55.919 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.917+0000 7f59beaf4640 10 monclient(hunting): picked mon.noname-c con 0x7f59b813f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:20:55.919 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.917+0000 7f59beaf4640 10 monclient(hunting): picked mon.noname-b con 0x7f59b81430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:20:55.919 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.917+0000 7f59beaf4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b8142520 con 0x7f59b80a98f0 2024-03-25T18:20:55.919 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.917+0000 7f59beaf4640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b80b28a0 con 0x7f59b813f710 2024-03-25T18:20:55.919 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.917+0000 7f59beaf4640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b80b2a30 con 0x7f59b81430c0 2024-03-25T18:20:55.919 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.917+0000 7f59beaf4640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:55.919 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.917+0000 7f59beaf4640 10 monclient(hunting): authenticate will time out at 7579.183140s 2024-03-25T18:20:55.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.930+0000 7f59bc890640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f59b80a98f0 0x7f59b80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43798/0 (socket says 172.21.15.122:43798) 2024-03-25T18:20:55.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.930+0000 7f59bc890640 1 -- 172.21.15.122:0/977204221 learned_addr learned my addr 172.21.15.122:0/977204221 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:20:55.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.944+0000 7f59b77fe640 1 -- 172.21.15.122:0/977204221 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2712420756 0 0) 0x7f59b8142520 con 0x7f59b80a98f0 2024-03-25T18:20:55.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.944+0000 7f59b77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:55.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.944+0000 7f59b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:55.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.944+0000 7f59b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:55.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.944+0000 7f59b77fe640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:20:55.946 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.944+0000 7f59b77fe640 1 -- 172.21.15.122:0/977204221 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a8003520 con 0x7f59b80a98f0 2024-03-25T18:20:55.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.946+0000 7f59b77fe640 1 -- 172.21.15.122:0/977204221 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286790803 0 0) 0x7f59b80b2a30 con 0x7f59b81430c0 2024-03-25T18:20:55.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.946+0000 7f59b77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:55.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.946+0000 7f59b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:55.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.946+0000 7f59b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:55.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.946+0000 7f59b77fe640 10 monclient(hunting): my global_id is 4403 2024-03-25T18:20:55.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.946+0000 7f59b77fe640 1 -- 172.21.15.122:0/977204221 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b8142520 con 0x7f59b81430c0 2024-03-25T18:20:55.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.958+0000 7f59b77fe640 1 -- 172.21.15.122:0/977204221 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4103562379 0 0) 0x7f59a8003520 con 0x7f59b80a98f0 2024-03-25T18:20:55.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.958+0000 7f59b77fe640 1 -- 172.21.15.122:0/977204221 >> v1:172.21.15.122:6790/0 conn(0x7f59b81430c0 legacy=0x7f59b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.958+0000 7f59b77fe640 1 -- 172.21.15.122:0/977204221 >> v1:172.21.15.204:6789/0 conn(0x7f59b813f710 legacy=0x7f59b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.958+0000 7f59b77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:55.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.958+0000 7f59b77fe640 1 monclient: found mon.noname-a 2024-03-25T18:20:55.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.958+0000 7f59b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:55.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.958+0000 7f59b77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:20:55.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.958+0000 7f59b77fe640 1 -- 172.21.15.122:0/977204221 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b80b44c0 con 0x7f59b80a98f0 2024-03-25T18:20:55.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.958+0000 7f59b77fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:55.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.958+0000 7f59b77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:55.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.958+0000 7f59b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:55.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.958+0000 7f59b77fe640 1 -- 172.21.15.122:0/977204221 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f59ac001690 con 0x7f59b80a98f0 2024-03-25T18:20:55.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.958+0000 7f59b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:55.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.958+0000 7f59b77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:55.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.958+0000 7f59b77fe640 10 monclient: dump: 2024-03-25T18:20:55.960 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:55.960 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:55.960 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:55.960 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:55.960 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:55.960 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:55.961 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:55.961 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:55.961 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:55.961 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:55.961 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.958+0000 7f59beaf4640 5 monclient: authenticate success, global_id 4431 2024-03-25T18:20:55.961 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.958+0000 7f59beaf4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:55.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.964+0000 7f59b77fe640 1 -- 172.21.15.122:0/977204221 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59ac0017f0 con 0x7f59b80a98f0 2024-03-25T18:20:55.966 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.964+0000 7f59b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:55.966 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.964+0000 7f59b77fe640 1 -- 172.21.15.122:0/977204221 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f59ac001c00 con 0x7f59b80a98f0 2024-03-25T18:20:55.966 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.964+0000 7f59beaf4640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:55.966 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.964+0000 7f59beaf4640 4 set_mon_vals no callback set 2024-03-25T18:20:55.966 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.964+0000 7f59b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:55.966 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.964+0000 7f59b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:55.966 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.964+0000 7f59b77fe640 10 monclient: dump: 2024-03-25T18:20:55.966 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:55.966 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:55.966 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:55.966 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:55.966 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:55.966 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:55.966 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:55.967 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:55.967 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:55.967 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:55.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.965+0000 7f59beaf4640 10 monclient: shutdown 2024-03-25T18:20:55.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.965+0000 7f59beaf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:55.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.965+0000 7f59beaf4640 1 -- 172.21.15.122:0/977204221 >> v1:172.21.15.122:6789/0 conn(0x7f59b80a98f0 legacy=0x7f59b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:55.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.965+0000 7f59beaf4640 1 -- 172.21.15.122:0/977204221 shutdown_connections 2024-03-25T18:20:55.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.965+0000 7f59beaf4640 1 -- 172.21.15.122:0/977204221 >> 172.21.15.122:0/977204221 conn(0x7f59b80b7ea0 msgr2=0x7f59b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:55.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.965+0000 7f59beaf4640 1 -- 172.21.15.122:0/977204221 shutdown_connections 2024-03-25T18:20:55.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.965+0000 7f59beaf4640 1 -- 172.21.15.122:0/977204221 wait complete. 2024-03-25T18:20:55.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.966+0000 7f59beaf4640 10 monclient: build_initial_monmap 2024-03-25T18:20:55.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.966+0000 7f59beaf4640 10 monclient: monmap: 2024-03-25T18:20:55.969 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:20:55.969 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:55.969 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:20:55.969 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:20:55.969 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:55.969 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:20:55.969 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:20:55.969 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:20:55.969 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:20:55.969 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:55.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.966+0000 7f59beaf4640 1 librados: starting msgr at 2024-03-25T18:20:55.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.966+0000 7f59beaf4640 1 librados: starting objecter 2024-03-25T18:20:55.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.967+0000 7f59beaf4640 1 Processor -- start 2024-03-25T18:20:55.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.967+0000 7f59beaf4640 1 -- start start 2024-03-25T18:20:55.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.967+0000 7f59beaf4640 1 librados: setting wanted keys 2024-03-25T18:20:55.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.967+0000 7f59beaf4640 1 librados: calling monclient init 2024-03-25T18:20:55.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.967+0000 7f59beaf4640 10 monclient: init 2024-03-25T18:20:55.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.968+0000 7f59beaf4640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:55.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.968+0000 7f59beaf4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:55.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.968+0000 7f59beaf4640 10 monclient(hunting): picked mon.noname-b con 0x7f59b80a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:20:55.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.968+0000 7f59beaf4640 10 monclient(hunting): picked mon.noname-c con 0x7f59b813f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:20:55.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.968+0000 7f59beaf4640 10 monclient(hunting): picked mon.noname-a con 0x7f59b81430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:20:55.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.968+0000 7f59beaf4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b80abd30 con 0x7f59b81430c0 2024-03-25T18:20:55.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.968+0000 7f59beaf4640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b81d77a0 con 0x7f59b80a98f0 2024-03-25T18:20:55.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.968+0000 7f59beaf4640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b81d51f0 con 0x7f59b813f710 2024-03-25T18:20:55.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.968+0000 7f59beaf4640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:55.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.968+0000 7f59beaf4640 10 monclient(hunting): authenticate will time out at 7579.234360s 2024-03-25T18:20:55.982 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.981+0000 7f59bd091640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f59b81430c0 0x7f59b81e1ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:43836/0 (socket says 172.21.15.122:43836) 2024-03-25T18:20:55.982 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.981+0000 7f59bd091640 1 -- 172.21.15.122:0/1605975916 learned_addr learned my addr 172.21.15.122:0/1605975916 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:20:55.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.986+0000 7f59b57fa640 10 client.?.objecter ms_handle_connect 0x7f59b81430c0 2024-03-25T18:20:55.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.986+0000 7f59b57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:55.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.991+0000 7f59b57fa640 10 client.?.objecter ms_handle_connect 0x7f59b813f710 2024-03-25T18:20:55.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.991+0000 7f59b57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:55.995 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.994+0000 7f59b57fa640 10 client.?.objecter ms_handle_connect 0x7f59b80a98f0 2024-03-25T18:20:55.995 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:55.994+0000 7f59b57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:56.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.006+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033434476 0 0) 0x7f59b81d51f0 con 0x7f59b813f710 2024-03-25T18:20:56.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.006+0000 7f59b57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:56.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.006+0000 7f59b57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:56.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.006+0000 7f59b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:56.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.006+0000 7f59b57fa640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:20:56.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.006+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f599c004eb0 con 0x7f59b813f710 2024-03-25T18:20:56.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.007+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1430437802 0 0) 0x7f59b81d77a0 con 0x7f59b80a98f0 2024-03-25T18:20:56.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.007+0000 7f59b57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:56.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.007+0000 7f59b57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:56.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.007+0000 7f59b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:56.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.007+0000 7f59b57fa640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:20:56.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.007+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b81d51f0 con 0x7f59b80a98f0 2024-03-25T18:20:56.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.021+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185514661 0 0) 0x7f59b80abd30 con 0x7f59b81430c0 2024-03-25T18:20:56.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.021+0000 7f59b57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:56.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.021+0000 7f59b57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:56.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.021+0000 7f59b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:56.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.021+0000 7f59b57fa640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:20:56.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.021+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59b81d77a0 con 0x7f59b81430c0 2024-03-25T18:20:56.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.031+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1246939819 0 0) 0x7f59b81d51f0 con 0x7f59b80a98f0 2024-03-25T18:20:56.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.031+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59b80abd30 con 0x7f59b80a98f0 2024-03-25T18:20:56.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.031+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f59ac001b00 con 0x7f59b80a98f0 2024-03-25T18:20:56.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.031+0000 7f59b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:56.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.031+0000 7f59b57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:56.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.031+0000 7f59b57fa640 10 monclient(hunting): dump: 2024-03-25T18:20:56.032 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:56.032 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:56.033 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:56.033 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:56.033 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:56.033 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:56.033 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:56.033 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:56.033 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:56.033 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:56.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.031+0000 7f59b57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:56.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.037+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 128402604 0 0) 0x7f599c004eb0 con 0x7f59b813f710 2024-03-25T18:20:56.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.037+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59b81d51f0 con 0x7f59b813f710 2024-03-25T18:20:56.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.037+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f59a4001930 con 0x7f59b813f710 2024-03-25T18:20:56.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.037+0000 7f59b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:56.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.037+0000 7f59b57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:56.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.037+0000 7f59b57fa640 10 monclient(hunting): dump: 2024-03-25T18:20:56.038 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:56.039 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:56.039 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:56.039 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:56.039 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:56.039 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:56.039 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:56.039 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:56.039 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:56.039 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:56.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.039+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 127934194 0 0) 0x7f59b81d77a0 con 0x7f59b81430c0 2024-03-25T18:20:56.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.039+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f599c004eb0 con 0x7f59b81430c0 2024-03-25T18:20:56.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.039+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f59b0001bd0 con 0x7f59b81430c0 2024-03-25T18:20:56.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.039+0000 7f59b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:56.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.039+0000 7f59b57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:56.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.039+0000 7f59b57fa640 10 monclient(hunting): dump: 2024-03-25T18:20:56.041 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:56.041 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:56.041 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:56.041 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:56.041 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:56.041 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:56.041 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:56.041 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:56.041 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:56.042 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:56.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.049+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4242764551 0 0) 0x7f59b80abd30 con 0x7f59b80a98f0 2024-03-25T18:20:56.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 >> v1:172.21.15.122:6790/0 conn(0x7f59b813f710 legacy=0x7f59b8138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 >> v1:172.21.15.122:6789/0 conn(0x7f59b81430c0 legacy=0x7f59b81e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59b57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:20:56.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59b57fa640 1 monclient: found mon.b 2024-03-25T18:20:56.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59b57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:56.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59b57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:56.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b81d83d0 con 0x7f59b80a98f0 2024-03-25T18:20:56.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59b57fa640 10 monclient: _finish_auth 0 2024-03-25T18:20:56.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59b57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:20:56.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:56.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59beaf4640 5 monclient: authenticate success, global_id 4429 2024-03-25T18:20:56.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59beaf4640 10 monclient: _renew_subs 2024-03-25T18:20:56.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59beaf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:56.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59beaf4640 1 -- 172.21.15.122:0/1605975916 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b8138a50 con 0x7f59b80a98f0 2024-03-25T18:20:56.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59beaf4640 10 client.4429.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:56.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59beaf4640 10 monclient: _renew_subs 2024-03-25T18:20:56.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59beaf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:56.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59beaf4640 1 -- 172.21.15.122:0/1605975916 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b81e35c0 con 0x7f59b80a98f0 2024-03-25T18:20:56.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.050+0000 7f59beaf4640 1 librados: init done 2024-03-25T18:20:56.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.051+0000 7f59beaf4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:56.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.051+0000 7f59beaf4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:56.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.051+0000 7f59beaf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:56.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.051+0000 7f59beaf4640 1 -- 172.21.15.122:0/1605975916 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b8074cc0 con 0x7f59b80a98f0 2024-03-25T18:20:56.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.332+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59ac004a90 con 0x7f59b80a98f0 2024-03-25T18:20:56.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.332+0000 7f59b57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:56.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.332+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f59ac001cc0 con 0x7f59b80a98f0 2024-03-25T18:20:56.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.332+0000 7f59b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:56.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.332+0000 7f59b57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:56.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.332+0000 7f59b57fa640 10 monclient: dump: 2024-03-25T18:20:56.333 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:20:56.334 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:20:56.334 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:56.334 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:20:56.334 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:56.334 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:20:56.334 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:20:56.334 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:20:56.334 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:20:56.334 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:20:56.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.332+0000 7f59b6ffd640 4 set_mon_vals no callback set 2024-03-25T18:20:56.335 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.332+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f59ac01ac10 con 0x7f59b80a98f0 2024-03-25T18:20:56.335 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.333+0000 7f59b7fff640 10 monclient: get_auth_request con 0x7f599c06e0f0 auth_method 0 2024-03-25T18:20:56.335 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.333+0000 7f59b57fa640 10 client.4429.objecter ms_handle_connect 0x7f599c06e0f0 2024-03-25T18:20:56.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.340+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f59ac054800 con 0x7f59b80a98f0 2024-03-25T18:20:56.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.340+0000 7f59b57fa640 10 client.4429.objecter ms_dispatch 0x7f59b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:56.342 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.340+0000 7f59b57fa640 3 client.4429.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:56.342 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.340+0000 7f59b57fa640 3 client.4429.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:56.342 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.340+0000 7f59b57fa640 20 client.4429.objecter dump_active .. 0 homeless 2024-03-25T18:20:56.342 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.341+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59ac050310 con 0x7f59b80a98f0 2024-03-25T18:20:56.342 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.341+0000 7f59b57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:56.342 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.341+0000 7f59b57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:56.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.583+0000 7f59beaf4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:56.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.583+0000 7f59beaf4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:56.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.583+0000 7f59beaf4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:20:56.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.583+0000 7f59beaf4640 1 -- 172.21.15.122:0/1605975916 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59b80627b0 con 0x7f59b80a98f0 2024-03-25T18:20:56.610 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.609+0000 7f59b57fa640 1 -- 172.21.15.122:0/1605975916 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f59ac052240 con 0x7f59b80a98f0 2024-03-25T18:20:56.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.609+0000 7f59b57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:56.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.609+0000 7f59b57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:56.611 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:20:56.611 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:56.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.612+0000 7f59beaf4640 10 librados: watch_flush enter 2024-03-25T18:20:56.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.612+0000 7f59beaf4640 10 librados: watch_flush exit 2024-03-25T18:20:56.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.612+0000 7f59beaf4640 20 client.4429.objecter shutdown clearing up homeless session... 2024-03-25T18:20:56.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.612+0000 7f59beaf4640 10 client.4429.objecter successfully canceled tick 2024-03-25T18:20:56.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.612+0000 7f59beaf4640 1 -- 172.21.15.122:0/1605975916 >> v1:172.21.15.204:6800/676228005 conn(0x7f599c06e0f0 legacy=0x7f599c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.612+0000 7f59beaf4640 10 monclient: shutdown 2024-03-25T18:20:56.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.612+0000 7f59beaf4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:56.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.612+0000 7f59beaf4640 1 -- 172.21.15.122:0/1605975916 >> v1:172.21.15.204:6789/0 conn(0x7f59b80a98f0 legacy=0x7f59b81d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.612+0000 7f59beaf4640 1 -- 172.21.15.122:0/1605975916 shutdown_connections 2024-03-25T18:20:56.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.612+0000 7f59beaf4640 1 -- 172.21.15.122:0/1605975916 >> 172.21.15.122:0/1605975916 conn(0x7f59b80b7ea0 msgr2=0x7f59b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:56.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.612+0000 7f59beaf4640 1 -- 172.21.15.122:0/1605975916 shutdown_connections 2024-03-25T18:20:56.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.612+0000 7f59beaf4640 1 -- 172.21.15.122:0/1605975916 wait complete. 2024-03-25T18:20:56.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:20:56.612+0000 7f59beaf4640 1 librados: shutdown 2024-03-25T18:20:56.625 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:03.431 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:03.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.570+0000 7fc1cc993640 10 monclient: get_monmap_and_config 2024-03-25T18:21:03.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.570+0000 7fc1cc993640 10 monclient: build_initial_monmap 2024-03-25T18:21:03.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.570+0000 7fc1cc993640 10 monclient: monmap: 2024-03-25T18:21:03.571 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:21:03.571 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:03.571 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:21:03.571178+0000 2024-03-25T18:21:03.571 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:21:03.571178+0000 2024-03-25T18:21:03.571 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:03.571 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:03.571 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:21:03.571 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:21:03.571 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:21:03.571 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:03.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.570+0000 7fc1cc993640 1 Processor -- start 2024-03-25T18:21:03.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.570+0000 7fc1cc993640 1 -- start start 2024-03-25T18:21:03.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.570+0000 7fc1cc993640 10 monclient: init 2024-03-25T18:21:03.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.571+0000 7fc1cc993640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:03.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.571+0000 7fc1cc993640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:21:03.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.571+0000 7fc1cc993640 10 monclient(hunting): picked mon.noname-a con 0x7fc1c80a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:21:03.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.571+0000 7fc1cc993640 10 monclient(hunting): picked mon.noname-c con 0x7fc1c813f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:21:03.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.571+0000 7fc1cc993640 10 monclient(hunting): picked mon.noname-b con 0x7fc1c81430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:21:03.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.571+0000 7fc1cc993640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c8142520 con 0x7fc1c80a98f0 2024-03-25T18:21:03.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.571+0000 7fc1cc993640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c80b28a0 con 0x7fc1c813f710 2024-03-25T18:21:03.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.571+0000 7fc1cc993640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c80b2a30 con 0x7fc1c81430c0 2024-03-25T18:21:03.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.571+0000 7fc1cc993640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:03.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.571+0000 7fc1cc993640 10 monclient(hunting): authenticate will time out at 7586.837710s 2024-03-25T18:21:03.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.574+0000 7fc1c6d9d640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7fc1c81430c0 0x7fc1c814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:35524/0 (socket says 172.21.15.122:35524) 2024-03-25T18:21:03.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.574+0000 7fc1c6d9d640 1 -- 172.21.15.122:0/909351829 learned_addr learned my addr 172.21.15.122:0/909351829 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:21:03.590 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.589+0000 7fc1c559a640 1 -- 172.21.15.122:0/909351829 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3289054942 0 0) 0x7fc1c80b2a30 con 0x7fc1c81430c0 2024-03-25T18:21:03.590 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.589+0000 7fc1c559a640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.591 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.589+0000 7fc1c559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.591 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.589+0000 7fc1c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.591 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.589+0000 7fc1c559a640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:21:03.591 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.589+0000 7fc1c559a640 1 -- 172.21.15.122:0/909351829 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1b0003520 con 0x7fc1c81430c0 2024-03-25T18:21:03.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.597+0000 7fc1c559a640 1 -- 172.21.15.122:0/909351829 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3991222380 0 0) 0x7fc1c80b28a0 con 0x7fc1c813f710 2024-03-25T18:21:03.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.597+0000 7fc1c559a640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.597+0000 7fc1c559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.597+0000 7fc1c559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.597+0000 7fc1c559a640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:21:03.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.597+0000 7fc1c559a640 1 -- 172.21.15.122:0/909351829 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1c80b2a30 con 0x7fc1c813f710 2024-03-25T18:21:03.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.599+0000 7fc1c559a640 1 -- 172.21.15.122:0/909351829 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3023241818 0 0) 0x7fc1b0003520 con 0x7fc1c81430c0 2024-03-25T18:21:03.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.599+0000 7fc1c559a640 1 -- 172.21.15.122:0/909351829 >> v1:172.21.15.204:6789/0 conn(0x7fc1c813f710 legacy=0x7fc1c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.599+0000 7fc1c559a640 1 -- 172.21.15.122:0/909351829 >> v1:172.21.15.122:6789/0 conn(0x7fc1c80a98f0 legacy=0x7fc1c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.599+0000 7fc1c559a640 10 monclient: _finish_hunting 0 2024-03-25T18:21:03.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.599+0000 7fc1c559a640 1 monclient: found mon.noname-b 2024-03-25T18:21:03.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.599+0000 7fc1c559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:03.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.599+0000 7fc1c559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:21:03.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.599+0000 7fc1c559a640 1 -- 172.21.15.122:0/909351829 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c80b44c0 con 0x7fc1c81430c0 2024-03-25T18:21:03.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.599+0000 7fc1c559a640 10 monclient: _finish_auth 0 2024-03-25T18:21:03.601 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.599+0000 7fc1c559a640 10 monclient: _check_auth_tickets 2024-03-25T18:21:03.601 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.599+0000 7fc1c559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:03.601 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.599+0000 7fc1cc993640 5 monclient: authenticate success, global_id 4409 2024-03-25T18:21:03.601 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.599+0000 7fc1cc993640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:03.602 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.601+0000 7fc1c559a640 1 -- 172.21.15.122:0/909351829 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fc1c00019d0 con 0x7fc1c81430c0 2024-03-25T18:21:03.602 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.601+0000 7fc1c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.602 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.601+0000 7fc1c559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:03.603 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.601+0000 7fc1c559a640 10 monclient: dump: 2024-03-25T18:21:03.603 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:03.603 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:03.603 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:03.603 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:03.603 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.603 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:03.603 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:03.603 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:03.603 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:03.603 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:03.603 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.601+0000 7fc1cc993640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:03.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.616+0000 7fc1c559a640 1 -- 172.21.15.122:0/909351829 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1c0001c20 con 0x7fc1c81430c0 2024-03-25T18:21:03.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.616+0000 7fc1c559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:03.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.616+0000 7fc1c559a640 1 -- 172.21.15.122:0/909351829 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fc1c0001fb0 con 0x7fc1c81430c0 2024-03-25T18:21:03.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.616+0000 7fc1cc993640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:03.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.616+0000 7fc1cc993640 4 set_mon_vals no callback set 2024-03-25T18:21:03.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.616+0000 7fc1c559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.616+0000 7fc1c559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:03.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.616+0000 7fc1c559a640 10 monclient: dump: 2024-03-25T18:21:03.617 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:03.617 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:03.618 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:03.618 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:03.618 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.618 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:03.618 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:03.618 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:03.618 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:03.618 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:03.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.616+0000 7fc1cc993640 10 monclient: shutdown 2024-03-25T18:21:03.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.616+0000 7fc1cc993640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:03.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.617+0000 7fc1cc993640 1 -- 172.21.15.122:0/909351829 >> v1:172.21.15.122:6790/0 conn(0x7fc1c81430c0 legacy=0x7fc1c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.617+0000 7fc1cc993640 1 -- 172.21.15.122:0/909351829 shutdown_connections 2024-03-25T18:21:03.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.617+0000 7fc1cc993640 1 -- 172.21.15.122:0/909351829 >> 172.21.15.122:0/909351829 conn(0x7fc1c80b7ea0 msgr2=0x7fc1c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:03.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.617+0000 7fc1cc993640 1 -- 172.21.15.122:0/909351829 shutdown_connections 2024-03-25T18:21:03.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.617+0000 7fc1cc993640 1 -- 172.21.15.122:0/909351829 wait complete. 2024-03-25T18:21:03.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.618+0000 7fc1cc993640 10 monclient: build_initial_monmap 2024-03-25T18:21:03.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.618+0000 7fc1cc993640 10 monclient: monmap: 2024-03-25T18:21:03.620 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:21:03.620 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:03.620 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:21:03.620 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:21:03.620 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:03.620 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:21:03.620 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:21:03.620 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:21:03.620 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:21:03.620 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:03.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.618+0000 7fc1cc993640 1 librados: starting msgr at 2024-03-25T18:21:03.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.618+0000 7fc1cc993640 1 librados: starting objecter 2024-03-25T18:21:03.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.618+0000 7fc1cc993640 1 Processor -- start 2024-03-25T18:21:03.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.618+0000 7fc1cc993640 1 -- start start 2024-03-25T18:21:03.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.618+0000 7fc1cc993640 1 librados: setting wanted keys 2024-03-25T18:21:03.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.618+0000 7fc1cc993640 1 librados: calling monclient init 2024-03-25T18:21:03.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.618+0000 7fc1cc993640 10 monclient: init 2024-03-25T18:21:03.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.619+0000 7fc1cc993640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:03.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.619+0000 7fc1cc993640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:03.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.619+0000 7fc1cc993640 10 monclient(hunting): picked mon.noname-b con 0x7fc1c80a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:21:03.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.619+0000 7fc1cc993640 10 monclient(hunting): picked mon.noname-c con 0x7fc1c813f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:21:03.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.619+0000 7fc1cc993640 10 monclient(hunting): picked mon.noname-a con 0x7fc1c81430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:21:03.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.619+0000 7fc1cc993640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c80b1870 con 0x7fc1c81430c0 2024-03-25T18:21:03.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.619+0000 7fc1cc993640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c81d7670 con 0x7fc1c80a98f0 2024-03-25T18:21:03.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.619+0000 7fc1cc993640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1c81d7950 con 0x7fc1c813f710 2024-03-25T18:21:03.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.619+0000 7fc1cc993640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:03.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.619+0000 7fc1cc993640 10 monclient(hunting): authenticate will time out at 7586.885723s 2024-03-25T18:21:03.633 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.632+0000 7fc1c659c640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fc1c80a98f0 0x7fc1c81381e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:59298/0 (socket says 172.21.15.122:59298) 2024-03-25T18:21:03.633 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.632+0000 7fc1c659c640 1 -- 172.21.15.122:0/395425922 learned_addr learned my addr 172.21.15.122:0/395425922 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:21:03.635 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.634+0000 7fc1b6ffd640 10 client.?.objecter ms_handle_connect 0x7fc1c813f710 2024-03-25T18:21:03.635 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.634+0000 7fc1b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:03.637 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.636+0000 7fc1b6ffd640 10 client.?.objecter ms_handle_connect 0x7fc1c81430c0 2024-03-25T18:21:03.637 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.636+0000 7fc1b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:03.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.644+0000 7fc1b6ffd640 10 client.?.objecter ms_handle_connect 0x7fc1c80a98f0 2024-03-25T18:21:03.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.644+0000 7fc1b6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:03.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.649+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586929174 0 0) 0x7fc1c81d7950 con 0x7fc1c813f710 2024-03-25T18:21:03.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.649+0000 7fc1b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.649+0000 7fc1b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.649+0000 7fc1b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.649+0000 7fc1b6ffd640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:21:03.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.649+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a8004eb0 con 0x7fc1c813f710 2024-03-25T18:21:03.665 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.664+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091681644 0 0) 0x7fc1c80b44c0 con 0x7fc1c81430c0 2024-03-25T18:21:03.665 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.664+0000 7fc1b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.664+0000 7fc1b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.664+0000 7fc1b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.664+0000 7fc1b6ffd640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:21:03.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.664+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1c81d7950 con 0x7fc1c81430c0 2024-03-25T18:21:03.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.665+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2397312151 0 0) 0x7fc1a8004eb0 con 0x7fc1c813f710 2024-03-25T18:21:03.666 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.665+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc1c80b44c0 con 0x7fc1c813f710 2024-03-25T18:21:03.667 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.666+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1321322113 0 0) 0x7fc1c81d7670 con 0x7fc1c80a98f0 2024-03-25T18:21:03.667 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.666+0000 7fc1b6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.667 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.666+0000 7fc1b6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.667 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.666+0000 7fc1b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.667 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.666+0000 7fc1b6ffd640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:21:03.667 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.666+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1a8004eb0 con 0x7fc1c80a98f0 2024-03-25T18:21:03.675 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.674+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fc1bc0019f0 con 0x7fc1c813f710 2024-03-25T18:21:03.675 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.674+0000 7fc1b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.675 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.674+0000 7fc1b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:03.675 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.674+0000 7fc1b6ffd640 10 monclient(hunting): dump: 2024-03-25T18:21:03.675 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:03.676 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:03.676 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:03.676 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:03.676 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.676 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:03.676 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:03.676 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:03.676 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:03.676 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:03.676 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.674+0000 7fc1b6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:03.679 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4287723307 0 0) 0x7fc1c80b44c0 con 0x7fc1c813f710 2024-03-25T18:21:03.679 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 >> v1:172.21.15.204:6789/0 conn(0x7fc1c80a98f0 legacy=0x7fc1c81381e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 >> v1:172.21.15.122:6789/0 conn(0x7fc1c81430c0 legacy=0x7fc1c81d4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1b6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:03.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1b6ffd640 1 monclient: found mon.c 2024-03-25T18:21:03.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:03.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1b6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:21:03.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1c81388f0 con 0x7fc1c813f710 2024-03-25T18:21:03.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1b6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:21:03.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1b6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:03.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:03.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1cc993640 5 monclient: authenticate success, global_id 4412 2024-03-25T18:21:03.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1cc993640 10 monclient: _renew_subs 2024-03-25T18:21:03.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1cc993640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:21:03.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1cc993640 1 -- 172.21.15.122:0/395425922 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c81d84f0 con 0x7fc1c813f710 2024-03-25T18:21:03.680 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1cc993640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:03.681 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1cc993640 10 monclient: _renew_subs 2024-03-25T18:21:03.681 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1cc993640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:21:03.681 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1cc993640 1 -- 172.21.15.122:0/395425922 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c81d5280 con 0x7fc1c813f710 2024-03-25T18:21:03.681 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.678+0000 7fc1cc993640 1 librados: init done 2024-03-25T18:21:03.681 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.679+0000 7fc1cc993640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:03.681 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.679+0000 7fc1cc993640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:03.681 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.679+0000 7fc1cc993640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:21:03.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.679+0000 7fc1cc993640 1 -- 172.21.15.122:0/395425922 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c80627b0 con 0x7fc1c813f710 2024-03-25T18:21:03.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.996+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc1bc003610 con 0x7fc1c813f710 2024-03-25T18:21:03.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.996+0000 7fc1b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:03.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.996+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fc1bc0039e0 con 0x7fc1c813f710 2024-03-25T18:21:03.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.996+0000 7fc1b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.996+0000 7fc1b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:03.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.996+0000 7fc1b6ffd640 10 monclient: dump: 2024-03-25T18:21:03.998 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:03.998 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:03.998 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:03.998 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:03.998 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.999 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:03.999 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:03.999 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:03.999 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:03.999 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:03.999 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.996+0000 7fc1c4d99640 4 set_mon_vals no callback set 2024-03-25T18:21:03.999 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.997+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fc1bc003c00 con 0x7fc1c813f710 2024-03-25T18:21:03.999 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.997+0000 7fc1c659c640 10 monclient: get_auth_request con 0x7fc1a806de20 auth_method 0 2024-03-25T18:21:03.999 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:03.998+0000 7fc1b6ffd640 10 client.4412.objecter ms_handle_connect 0x7fc1a806de20 2024-03-25T18:21:04.006 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.004+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fc1bc053620 con 0x7fc1c813f710 2024-03-25T18:21:04.006 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.004+0000 7fc1b6ffd640 10 client.4412.objecter ms_dispatch 0x7fc1c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:04.006 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.004+0000 7fc1b6ffd640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:04.006 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.004+0000 7fc1b6ffd640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:04.006 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.005+0000 7fc1b6ffd640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-25T18:21:04.006 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.005+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc1bc04f130 con 0x7fc1c813f710 2024-03-25T18:21:04.006 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.005+0000 7fc1b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:04.006 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.005+0000 7fc1b6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:04.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.252+0000 7fc1cc993640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:04.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.252+0000 7fc1cc993640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:04.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.252+0000 7fc1cc993640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:21:04.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.252+0000 7fc1cc993640 1 -- 172.21.15.122:0/395425922 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc1c81d7b00 con 0x7fc1c813f710 2024-03-25T18:21:04.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.297+0000 7fc1b6ffd640 1 -- 172.21.15.122:0/395425922 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fc1c81d7b00 con 0x7fc1c813f710 2024-03-25T18:21:04.299 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.297+0000 7fc1b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:04.299 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.297+0000 7fc1b6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:04.299 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:21:04.299 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:04.301 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.299+0000 7fc1cc993640 10 librados: watch_flush enter 2024-03-25T18:21:04.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.300+0000 7fc1cc993640 10 librados: watch_flush exit 2024-03-25T18:21:04.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.300+0000 7fc1cc993640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-25T18:21:04.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.300+0000 7fc1cc993640 10 client.4412.objecter successfully canceled tick 2024-03-25T18:21:04.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.300+0000 7fc1cc993640 1 -- 172.21.15.122:0/395425922 >> v1:172.21.15.204:6800/676228005 conn(0x7fc1a806de20 legacy=0x7fc1a80702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.300+0000 7fc1cc993640 10 monclient: shutdown 2024-03-25T18:21:04.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.300+0000 7fc1cc993640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:04.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.300+0000 7fc1cc993640 1 -- 172.21.15.122:0/395425922 >> v1:172.21.15.122:6790/0 conn(0x7fc1c813f710 legacy=0x7fc1c81d7de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.300+0000 7fc1cc993640 1 -- 172.21.15.122:0/395425922 shutdown_connections 2024-03-25T18:21:04.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.300+0000 7fc1cc993640 1 -- 172.21.15.122:0/395425922 >> 172.21.15.122:0/395425922 conn(0x7fc1c80b7ea0 msgr2=0x7fc1c8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:04.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.300+0000 7fc1cc993640 1 -- 172.21.15.122:0/395425922 shutdown_connections 2024-03-25T18:21:04.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.300+0000 7fc1cc993640 1 -- 172.21.15.122:0/395425922 wait complete. 2024-03-25T18:21:04.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:04.300+0000 7fc1cc993640 1 librados: shutdown 2024-03-25T18:21:04.312 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:06.274 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:05:00.016 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:06.280 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:05:00.023 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:06.319 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:05:00.062 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:06.341 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:05:00.087 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:06.349 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:05:00.094 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:06.357 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:05:00.099 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:06.364 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:05:00.106 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:06.382 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:05:00.127 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:06.392 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:05:00.135 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:11.119 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:11.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.233+0000 7fd423298640 10 monclient: get_monmap_and_config 2024-03-25T18:21:11.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.233+0000 7fd423298640 10 monclient: build_initial_monmap 2024-03-25T18:21:11.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.233+0000 7fd423298640 10 monclient: monmap: 2024-03-25T18:21:11.235 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:21:11.235 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:11.235 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:21:11.234606+0000 2024-03-25T18:21:11.235 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:21:11.234606+0000 2024-03-25T18:21:11.235 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:11.235 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:11.235 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:21:11.235 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:21:11.235 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:21:11.235 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:11.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.234+0000 7fd423298640 1 Processor -- start 2024-03-25T18:21:11.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.234+0000 7fd423298640 1 -- start start 2024-03-25T18:21:11.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.234+0000 7fd423298640 10 monclient: init 2024-03-25T18:21:11.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.235+0000 7fd423298640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:11.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.235+0000 7fd423298640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:21:11.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.235+0000 7fd423298640 10 monclient(hunting): picked mon.noname-c con 0x7fd41c0a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:21:11.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.235+0000 7fd423298640 10 monclient(hunting): picked mon.noname-b con 0x7fd41c13f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:21:11.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.235+0000 7fd423298640 10 monclient(hunting): picked mon.noname-a con 0x7fd41c1430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:21:11.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.235+0000 7fd423298640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd41c142520 con 0x7fd41c1430c0 2024-03-25T18:21:11.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.235+0000 7fd423298640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd41c0b28a0 con 0x7fd41c0a98f0 2024-03-25T18:21:11.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.235+0000 7fd423298640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd41c0b2a30 con 0x7fd41c13f710 2024-03-25T18:21:11.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.235+0000 7fd423298640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:11.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.235+0000 7fd423298640 10 monclient(hunting): authenticate will time out at 7594.501257s 2024-03-25T18:21:11.244 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.243+0000 7fd421034640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fd41c0a98f0 0x7fd41c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:40058/0 (socket says 172.21.15.122:40058) 2024-03-25T18:21:11.244 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.243+0000 7fd421034640 1 -- 172.21.15.122:0/2650665345 learned_addr learned my addr 172.21.15.122:0/2650665345 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:21:11.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.259+0000 7fd413fff640 1 -- 172.21.15.122:0/2650665345 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836414492 0 0) 0x7fd41c0b28a0 con 0x7fd41c0a98f0 2024-03-25T18:21:11.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.260+0000 7fd413fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:11.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.260+0000 7fd413fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:11.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.260+0000 7fd413fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:11.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.260+0000 7fd413fff640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:21:11.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.260+0000 7fd413fff640 1 -- 172.21.15.122:0/2650665345 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd408003520 con 0x7fd41c0a98f0 2024-03-25T18:21:11.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.266+0000 7fd413fff640 1 -- 172.21.15.122:0/2650665345 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927664619 0 0) 0x7fd41c142520 con 0x7fd41c1430c0 2024-03-25T18:21:11.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.266+0000 7fd413fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:11.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.266+0000 7fd413fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:11.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.266+0000 7fd413fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:11.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.266+0000 7fd413fff640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:21:11.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.266+0000 7fd413fff640 1 -- 172.21.15.122:0/2650665345 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd41c0b28a0 con 0x7fd41c1430c0 2024-03-25T18:21:11.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd413fff640 1 -- 172.21.15.122:0/2650665345 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1314695656 0 0) 0x7fd408003520 con 0x7fd41c0a98f0 2024-03-25T18:21:11.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd413fff640 1 -- 172.21.15.122:0/2650665345 >> v1:172.21.15.122:6790/0 conn(0x7fd41c13f710 legacy=0x7fd41c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd413fff640 1 -- 172.21.15.122:0/2650665345 >> v1:172.21.15.122:6789/0 conn(0x7fd41c1430c0 legacy=0x7fd41c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd413fff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:11.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd413fff640 1 monclient: found mon.noname-c 2024-03-25T18:21:11.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd413fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:11.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd413fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:21:11.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd413fff640 1 -- 172.21.15.122:0/2650665345 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd41c0b44c0 con 0x7fd41c0a98f0 2024-03-25T18:21:11.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd413fff640 10 monclient: _finish_auth 0 2024-03-25T18:21:11.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd413fff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:11.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd413fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:11.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd413fff640 1 -- 172.21.15.122:0/2650665345 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd418001690 con 0x7fd41c0a98f0 2024-03-25T18:21:11.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd423298640 5 monclient: authenticate success, global_id 4438 2024-03-25T18:21:11.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd423298640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:11.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd413fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:11.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd413fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:11.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd413fff640 10 monclient: dump: 2024-03-25T18:21:11.280 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:11.280 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:11.280 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:11.280 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:11.280 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:11.280 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:11.281 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:11.281 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:11.281 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:11.281 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:11.281 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.278+0000 7fd423298640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:11.294 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.293+0000 7fd413fff640 1 -- 172.21.15.122:0/2650665345 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd4180017f0 con 0x7fd41c0a98f0 2024-03-25T18:21:11.294 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.293+0000 7fd413fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:11.294 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.293+0000 7fd413fff640 1 -- 172.21.15.122:0/2650665345 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd418001c00 con 0x7fd41c0a98f0 2024-03-25T18:21:11.294 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.293+0000 7fd413fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:11.295 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.293+0000 7fd413fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:11.295 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.293+0000 7fd413fff640 10 monclient: dump: 2024-03-25T18:21:11.295 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:11.295 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:11.295 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:11.295 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:11.295 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:11.295 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:11.295 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:11.295 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:11.295 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:11.295 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:11.295 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.293+0000 7fd423298640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:11.296 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.293+0000 7fd423298640 4 set_mon_vals no callback set 2024-03-25T18:21:11.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.294+0000 7fd423298640 10 monclient: shutdown 2024-03-25T18:21:11.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.294+0000 7fd423298640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:11.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.294+0000 7fd423298640 1 -- 172.21.15.122:0/2650665345 >> v1:172.21.15.204:6789/0 conn(0x7fd41c0a98f0 legacy=0x7fd41c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.294+0000 7fd423298640 1 -- 172.21.15.122:0/2650665345 shutdown_connections 2024-03-25T18:21:11.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.294+0000 7fd423298640 1 -- 172.21.15.122:0/2650665345 >> 172.21.15.122:0/2650665345 conn(0x7fd41c0b7ea0 msgr2=0x7fd41c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:11.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.294+0000 7fd423298640 1 -- 172.21.15.122:0/2650665345 shutdown_connections 2024-03-25T18:21:11.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.294+0000 7fd423298640 1 -- 172.21.15.122:0/2650665345 wait complete. 2024-03-25T18:21:11.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.294+0000 7fd423298640 10 monclient: build_initial_monmap 2024-03-25T18:21:11.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.295+0000 7fd423298640 10 monclient: monmap: 2024-03-25T18:21:11.298 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:21:11.298 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:11.298 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:21:11.298 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:21:11.299 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:11.299 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:21:11.299 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:21:11.299 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:21:11.299 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:21:11.299 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:11.299 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.295+0000 7fd423298640 1 librados: starting msgr at 2024-03-25T18:21:11.299 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.295+0000 7fd423298640 1 librados: starting objecter 2024-03-25T18:21:11.299 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.295+0000 7fd423298640 1 Processor -- start 2024-03-25T18:21:11.299 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.295+0000 7fd423298640 1 -- start start 2024-03-25T18:21:11.299 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.295+0000 7fd423298640 1 librados: setting wanted keys 2024-03-25T18:21:11.299 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.295+0000 7fd423298640 1 librados: calling monclient init 2024-03-25T18:21:11.299 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.295+0000 7fd423298640 10 monclient: init 2024-03-25T18:21:11.299 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.296+0000 7fd423298640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:11.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.296+0000 7fd423298640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:11.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.296+0000 7fd423298640 10 monclient(hunting): picked mon.noname-c con 0x7fd41c0a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:21:11.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.296+0000 7fd423298640 10 monclient(hunting): picked mon.noname-a con 0x7fd41c13f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:21:11.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.296+0000 7fd423298640 10 monclient(hunting): picked mon.noname-b con 0x7fd41c1430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:21:11.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.296+0000 7fd423298640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd41c138250 con 0x7fd41c13f710 2024-03-25T18:21:11.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.296+0000 7fd423298640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd41c137d50 con 0x7fd41c1430c0 2024-03-25T18:21:11.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.296+0000 7fd423298640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd41c137ee0 con 0x7fd41c0a98f0 2024-03-25T18:21:11.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.296+0000 7fd423298640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:11.300 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.296+0000 7fd423298640 10 monclient(hunting): authenticate will time out at 7594.562316s 2024-03-25T18:21:11.305 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.304+0000 7fd421835640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fd41c1430c0 0x7fd41c1d7df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:40114/0 (socket says 172.21.15.122:40114) 2024-03-25T18:21:11.305 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.304+0000 7fd421835640 1 -- 172.21.15.122:0/2349969803 learned_addr learned my addr 172.21.15.122:0/2349969803 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:21:11.308 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.307+0000 7fd411ffb640 10 client.?.objecter ms_handle_connect 0x7fd41c1430c0 2024-03-25T18:21:11.308 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.307+0000 7fd411ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:11.308 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.307+0000 7fd411ffb640 10 client.?.objecter ms_handle_connect 0x7fd41c0a98f0 2024-03-25T18:21:11.308 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.307+0000 7fd411ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:11.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.317+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807845820 0 0) 0x7fd41c137d50 con 0x7fd41c1430c0 2024-03-25T18:21:11.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.317+0000 7fd411ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:11.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.317+0000 7fd411ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:11.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.317+0000 7fd411ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:11.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.317+0000 7fd411ffb640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:21:11.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.317+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd400004eb0 con 0x7fd41c1430c0 2024-03-25T18:21:11.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.319+0000 7fd411ffb640 10 client.?.objecter ms_handle_connect 0x7fd41c13f710 2024-03-25T18:21:11.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.319+0000 7fd411ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:11.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.319+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2700606798 0 0) 0x7fd41c137ee0 con 0x7fd41c0a98f0 2024-03-25T18:21:11.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.319+0000 7fd411ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:11.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.319+0000 7fd411ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:11.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.319+0000 7fd411ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:11.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.319+0000 7fd411ffb640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:21:11.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.319+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd41c137d50 con 0x7fd41c0a98f0 2024-03-25T18:21:11.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.327+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3670096115 0 0) 0x7fd400004eb0 con 0x7fd41c1430c0 2024-03-25T18:21:11.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.327+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd41c137ee0 con 0x7fd41c1430c0 2024-03-25T18:21:11.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.327+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd414001c90 con 0x7fd41c1430c0 2024-03-25T18:21:11.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.327+0000 7fd411ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:11.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.327+0000 7fd411ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:11.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.327+0000 7fd411ffb640 10 monclient(hunting): dump: 2024-03-25T18:21:11.329 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:11.329 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:11.329 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:11.329 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:11.329 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:11.329 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:11.329 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:11.329 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:11.329 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:11.329 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:11.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.327+0000 7fd411ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:11.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.328+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341240532 0 0) 0x7fd41c138250 con 0x7fd41c13f710 2024-03-25T18:21:11.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.328+0000 7fd411ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:11.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.328+0000 7fd411ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:11.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.328+0000 7fd411ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:11.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.328+0000 7fd411ffb640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:21:11.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.328+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd400004eb0 con 0x7fd41c13f710 2024-03-25T18:21:11.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.334+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3619246610 0 0) 0x7fd41c137ee0 con 0x7fd41c1430c0 2024-03-25T18:21:11.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.334+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 >> v1:172.21.15.122:6790/0 conn(0x7fd41c0a98f0 legacy=0x7fd41c137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 >> v1:172.21.15.122:6789/0 conn(0x7fd41c13f710 legacy=0x7fd41c1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.337 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd411ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:21:11.337 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd411ffb640 1 monclient: found mon.b 2024-03-25T18:21:11.337 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd411ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:11.337 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd411ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:11.337 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd41c0b7730 con 0x7fd41c1430c0 2024-03-25T18:21:11.337 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd411ffb640 10 monclient: _finish_auth 0 2024-03-25T18:21:11.337 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd411ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:21:11.337 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd411ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:11.337 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd423298640 5 monclient: authenticate success, global_id 4441 2024-03-25T18:21:11.337 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd423298640 10 monclient: _renew_subs 2024-03-25T18:21:11.337 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd423298640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:11.337 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd423298640 1 -- 172.21.15.122:0/2349969803 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd41c0b7920 con 0x7fd41c1430c0 2024-03-25T18:21:11.337 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd423298640 10 client.4441.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:11.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd423298640 10 monclient: _renew_subs 2024-03-25T18:21:11.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd423298640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:11.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd423298640 1 -- 172.21.15.122:0/2349969803 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd41c1e33d0 con 0x7fd41c1430c0 2024-03-25T18:21:11.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.335+0000 7fd423298640 1 librados: init done 2024-03-25T18:21:11.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.336+0000 7fd423298640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:11.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.336+0000 7fd423298640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:11.339 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.336+0000 7fd423298640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:11.339 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.336+0000 7fd423298640 1 -- 172.21.15.122:0/2349969803 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd41c0627b0 con 0x7fd41c1430c0 2024-03-25T18:21:11.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.612+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd414003990 con 0x7fd41c1430c0 2024-03-25T18:21:11.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.612+0000 7fd411ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:11.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.612+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd414003d60 con 0x7fd41c1430c0 2024-03-25T18:21:11.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.612+0000 7fd411ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:11.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.612+0000 7fd411ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:11.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.612+0000 7fd411ffb640 10 monclient: dump: 2024-03-25T18:21:11.614 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:11.615 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:11.615 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:11.615 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:11.615 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:11.615 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:11.615 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:11.615 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:11.615 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:11.615 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:11.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.612+0000 7fd4137fe640 4 set_mon_vals no callback set 2024-03-25T18:21:11.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.613+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fd414003f80 con 0x7fd41c1430c0 2024-03-25T18:21:11.616 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.614+0000 7fd421034640 10 monclient: get_auth_request con 0x7fd40006de00 auth_method 0 2024-03-25T18:21:11.616 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.614+0000 7fd411ffb640 10 client.4441.objecter ms_handle_connect 0x7fd40006de00 2024-03-25T18:21:11.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.620+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fd414054ad0 con 0x7fd41c1430c0 2024-03-25T18:21:11.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.620+0000 7fd411ffb640 10 client.4441.objecter ms_dispatch 0x7fd41c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:11.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.620+0000 7fd411ffb640 3 client.4441.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:11.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.620+0000 7fd411ffb640 3 client.4441.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:11.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.620+0000 7fd411ffb640 20 client.4441.objecter dump_active .. 0 homeless 2024-03-25T18:21:11.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.621+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd4140505e0 con 0x7fd41c1430c0 2024-03-25T18:21:11.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.621+0000 7fd411ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:11.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.621+0000 7fd411ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:11.868 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.867+0000 7fd423298640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:11.868 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.867+0000 7fd423298640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:11.868 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.867+0000 7fd423298640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:11.868 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.867+0000 7fd423298640 1 -- 172.21.15.122:0/2349969803 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd41c138400 con 0x7fd41c1430c0 2024-03-25T18:21:11.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.892+0000 7fd411ffb640 1 -- 172.21.15.122:0/2349969803 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fd41c138400 con 0x7fd41c1430c0 2024-03-25T18:21:11.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.893+0000 7fd411ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:11.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.893+0000 7fd411ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:11.894 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:21:11.894 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:11.897 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.895+0000 7fd423298640 10 librados: watch_flush enter 2024-03-25T18:21:11.897 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.895+0000 7fd423298640 10 librados: watch_flush exit 2024-03-25T18:21:11.897 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.895+0000 7fd423298640 20 client.4441.objecter shutdown clearing up homeless session... 2024-03-25T18:21:11.897 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.895+0000 7fd423298640 10 client.4441.objecter successfully canceled tick 2024-03-25T18:21:11.897 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.895+0000 7fd423298640 1 -- 172.21.15.122:0/2349969803 >> v1:172.21.15.204:6800/676228005 conn(0x7fd40006de00 legacy=0x7fd4000702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.897 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.895+0000 7fd423298640 10 monclient: shutdown 2024-03-25T18:21:11.897 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.895+0000 7fd423298640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:11.897 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.895+0000 7fd423298640 1 -- 172.21.15.122:0/2349969803 >> v1:172.21.15.204:6789/0 conn(0x7fd41c1430c0 legacy=0x7fd41c1d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.897 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.895+0000 7fd423298640 1 -- 172.21.15.122:0/2349969803 shutdown_connections 2024-03-25T18:21:11.898 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.895+0000 7fd423298640 1 -- 172.21.15.122:0/2349969803 >> 172.21.15.122:0/2349969803 conn(0x7fd41c0b7ea0 msgr2=0x7fd41c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:11.898 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.895+0000 7fd423298640 1 -- 172.21.15.122:0/2349969803 shutdown_connections 2024-03-25T18:21:11.898 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.895+0000 7fd423298640 1 -- 172.21.15.122:0/2349969803 wait complete. 2024-03-25T18:21:11.898 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:11.895+0000 7fd423298640 1 librados: shutdown 2024-03-25T18:21:11.909 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:18.716 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:18.815 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.814+0000 7f90152f6640 10 monclient: get_monmap_and_config 2024-03-25T18:21:18.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.814+0000 7f90152f6640 10 monclient: build_initial_monmap 2024-03-25T18:21:18.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.814+0000 7f90152f6640 10 monclient: monmap: 2024-03-25T18:21:18.816 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:21:18.816 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:18.816 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:21:18.815743+0000 2024-03-25T18:21:18.816 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:21:18.815743+0000 2024-03-25T18:21:18.816 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:18.816 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:18.816 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:21:18.816 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:21:18.816 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:21:18.816 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:18.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.815+0000 7f90152f6640 1 Processor -- start 2024-03-25T18:21:18.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.815+0000 7f90152f6640 1 -- start start 2024-03-25T18:21:18.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.815+0000 7f90152f6640 10 monclient: init 2024-03-25T18:21:18.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.815+0000 7f90152f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:18.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.815+0000 7f90152f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:18.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.816+0000 7f90152f6640 10 monclient(hunting): picked mon.noname-b con 0x7f90100a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:21:18.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.816+0000 7f90152f6640 10 monclient(hunting): picked mon.noname-c con 0x7f901013f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:21:18.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.816+0000 7f90152f6640 10 monclient(hunting): picked mon.noname-a con 0x7f90101430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:21:18.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.816+0000 7f90152f6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9010142520 con 0x7f90101430c0 2024-03-25T18:21:18.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.816+0000 7f90152f6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90100b28a0 con 0x7f901013f710 2024-03-25T18:21:18.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.816+0000 7f90152f6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90100b2a30 con 0x7f90100a98f0 2024-03-25T18:21:18.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.816+0000 7f90152f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:18.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.816+0000 7f90152f6640 10 monclient(hunting): authenticate will time out at 7602.082169s 2024-03-25T18:21:18.829 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.828+0000 7f900e7fc640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f901013f710 0x7f90101429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:44834/0 (socket says 172.21.15.122:44834) 2024-03-25T18:21:18.829 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.828+0000 7f900e7fc640 1 -- 172.21.15.122:0/2922215738 learned_addr learned my addr 172.21.15.122:0/2922215738 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:21:18.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.891+0000 7f900dffb640 1 -- 172.21.15.122:0/2922215738 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430411350 0 0) 0x7f9010142520 con 0x7f90101430c0 2024-03-25T18:21:18.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.891+0000 7f900dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:18.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.891+0000 7f900dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:18.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.891+0000 7f900dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:18.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.891+0000 7f900dffb640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:21:18.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.891+0000 7f900dffb640 1 -- 172.21.15.122:0/2922215738 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ff8003520 con 0x7f90101430c0 2024-03-25T18:21:18.901 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.900+0000 7f900dffb640 1 -- 172.21.15.122:0/2922215738 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6421659 0 0) 0x7f90100b2a30 con 0x7f90100a98f0 2024-03-25T18:21:18.901 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.900+0000 7f900dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:18.901 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.900+0000 7f900dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:18.901 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.900+0000 7f900dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:18.901 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.900+0000 7f900dffb640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:21:18.902 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.900+0000 7f900dffb640 1 -- 172.21.15.122:0/2922215738 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9010142520 con 0x7f90100a98f0 2024-03-25T18:21:18.902 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.901+0000 7f900dffb640 1 -- 172.21.15.122:0/2922215738 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764488773 0 0) 0x7f90100b28a0 con 0x7f901013f710 2024-03-25T18:21:18.902 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.901+0000 7f900dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:18.902 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.901+0000 7f900dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:18.902 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.901+0000 7f900dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:18.902 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.901+0000 7f900dffb640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:21:18.902 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.901+0000 7f900dffb640 1 -- 172.21.15.122:0/2922215738 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90100b2a30 con 0x7f901013f710 2024-03-25T18:21:18.915 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.914+0000 7f900dffb640 1 -- 172.21.15.122:0/2922215738 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2015329566 0 0) 0x7f8ff8003520 con 0x7f90101430c0 2024-03-25T18:21:18.915 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.914+0000 7f900dffb640 1 -- 172.21.15.122:0/2922215738 >> v1:172.21.15.122:6790/0 conn(0x7f90100a98f0 legacy=0x7f90100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.915 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.914+0000 7f900dffb640 1 -- 172.21.15.122:0/2922215738 >> v1:172.21.15.204:6789/0 conn(0x7f901013f710 legacy=0x7f90101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.915 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.914+0000 7f900dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:21:18.915 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.914+0000 7f900dffb640 1 monclient: found mon.noname-a 2024-03-25T18:21:18.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.914+0000 7f900dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:18.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.914+0000 7f900dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:21:18.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.914+0000 7f900dffb640 1 -- 172.21.15.122:0/2922215738 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90100b44c0 con 0x7f90101430c0 2024-03-25T18:21:18.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.914+0000 7f900dffb640 10 monclient: _finish_auth 0 2024-03-25T18:21:18.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.914+0000 7f900dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:21:18.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.914+0000 7f900dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:18.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.914+0000 7f900dffb640 1 -- 172.21.15.122:0/2922215738 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9008001a10 con 0x7f90101430c0 2024-03-25T18:21:18.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.914+0000 7f900dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:18.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.914+0000 7f900dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:18.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.914+0000 7f900dffb640 10 monclient: dump: 2024-03-25T18:21:18.916 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:18.916 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:18.916 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:18.916 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:18.916 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:18.916 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:18.916 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:18.917 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:18.917 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:18.917 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:18.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.914+0000 7f90152f6640 5 monclient: authenticate success, global_id 4449 2024-03-25T18:21:18.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.914+0000 7f90152f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:18.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.929+0000 7f900dffb640 1 -- 172.21.15.122:0/2922215738 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9008001b70 con 0x7f90101430c0 2024-03-25T18:21:18.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.929+0000 7f900dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:18.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.929+0000 7f900dffb640 1 -- 172.21.15.122:0/2922215738 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9008001f80 con 0x7f90101430c0 2024-03-25T18:21:18.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.929+0000 7f90152f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:18.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.929+0000 7f90152f6640 4 set_mon_vals no callback set 2024-03-25T18:21:18.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.929+0000 7f900dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:18.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.929+0000 7f900dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:18.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.929+0000 7f900dffb640 10 monclient: dump: 2024-03-25T18:21:18.931 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:18.931 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:18.931 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:18.931 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:18.931 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:18.931 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:18.932 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:18.932 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:18.932 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:18.932 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:18.933 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.930+0000 7f90152f6640 10 monclient: shutdown 2024-03-25T18:21:18.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.930+0000 7f90152f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:18.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.930+0000 7f90152f6640 1 -- 172.21.15.122:0/2922215738 >> v1:172.21.15.122:6789/0 conn(0x7f90101430c0 legacy=0x7f901014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.930+0000 7f90152f6640 1 -- 172.21.15.122:0/2922215738 shutdown_connections 2024-03-25T18:21:18.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.930+0000 7f90152f6640 1 -- 172.21.15.122:0/2922215738 >> 172.21.15.122:0/2922215738 conn(0x7f90100b7ea0 msgr2=0x7f90100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:18.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.930+0000 7f90152f6640 1 -- 172.21.15.122:0/2922215738 shutdown_connections 2024-03-25T18:21:18.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.930+0000 7f90152f6640 1 -- 172.21.15.122:0/2922215738 wait complete. 2024-03-25T18:21:18.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.931+0000 7f90152f6640 10 monclient: build_initial_monmap 2024-03-25T18:21:18.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.931+0000 7f90152f6640 10 monclient: monmap: 2024-03-25T18:21:18.934 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:21:18.934 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:18.934 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:21:18.934 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:21:18.934 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:18.934 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:21:18.935 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:21:18.935 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:21:18.935 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:21:18.935 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:18.935 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.931+0000 7f90152f6640 1 librados: starting msgr at 2024-03-25T18:21:18.935 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.931+0000 7f90152f6640 1 librados: starting objecter 2024-03-25T18:21:18.935 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.931+0000 7f90152f6640 1 Processor -- start 2024-03-25T18:21:18.935 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.932+0000 7f90152f6640 1 -- start start 2024-03-25T18:21:18.935 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.932+0000 7f90152f6640 1 librados: setting wanted keys 2024-03-25T18:21:18.935 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.932+0000 7f90152f6640 1 librados: calling monclient init 2024-03-25T18:21:18.935 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.932+0000 7f90152f6640 10 monclient: init 2024-03-25T18:21:18.935 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.932+0000 7f90152f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:18.935 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.932+0000 7f90152f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:21:18.935 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.932+0000 7f90152f6640 10 monclient(hunting): picked mon.noname-a con 0x7f90100a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:21:18.935 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.932+0000 7f90152f6640 10 monclient(hunting): picked mon.noname-b con 0x7f901013f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:21:18.935 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.932+0000 7f90152f6640 10 monclient(hunting): picked mon.noname-c con 0x7f90101430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:21:18.935 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.932+0000 7f90152f6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90100abb00 con 0x7f90100a98f0 2024-03-25T18:21:18.936 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.932+0000 7f90152f6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90100abde0 con 0x7f901013f710 2024-03-25T18:21:18.936 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.932+0000 7f90152f6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9010138e20 con 0x7f90101430c0 2024-03-25T18:21:18.936 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.932+0000 7f90152f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:18.936 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.932+0000 7f90152f6640 10 monclient(hunting): authenticate will time out at 7602.198597s 2024-03-25T18:21:18.943 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.942+0000 7f900effd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f90100a98f0 0x7f90101385e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:48662/0 (socket says 172.21.15.122:48662) 2024-03-25T18:21:18.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.943+0000 7f900effd640 1 -- 172.21.15.122:0/4081951894 learned_addr learned my addr 172.21.15.122:0/4081951894 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:21:18.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.943+0000 7f8ff77fe640 10 client.?.objecter ms_handle_connect 0x7f90100a98f0 2024-03-25T18:21:18.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.943+0000 7f8ff77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:18.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.943+0000 7f8ff77fe640 10 client.?.objecter ms_handle_connect 0x7f901013f710 2024-03-25T18:21:18.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.943+0000 7f8ff77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:18.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.947+0000 7f8ff77fe640 10 client.?.objecter ms_handle_connect 0x7f90101430c0 2024-03-25T18:21:18.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.947+0000 7f8ff77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:18.955 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.954+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 901950196 0 0) 0x7f90100abde0 con 0x7f901013f710 2024-03-25T18:21:18.955 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.954+0000 7f8ff77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:18.955 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.954+0000 7f8ff77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:18.955 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.954+0000 7f8ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:18.955 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.954+0000 7f8ff77fe640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:21:18.956 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.954+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ffc004eb0 con 0x7f901013f710 2024-03-25T18:21:18.957 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.956+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050686843 0 0) 0x7f90100abb00 con 0x7f90100a98f0 2024-03-25T18:21:18.957 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.956+0000 7f8ff77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:18.957 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.956+0000 7f8ff77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:18.957 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.956+0000 7f8ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:18.957 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.956+0000 7f8ff77fe640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:21:18.958 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.956+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90100abde0 con 0x7f90100a98f0 2024-03-25T18:21:18.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.963+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1437156611 0 0) 0x7f9010138e20 con 0x7f90101430c0 2024-03-25T18:21:18.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.963+0000 7f8ff77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:18.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.963+0000 7f8ff77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:18.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.963+0000 7f8ff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:18.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.963+0000 7f8ff77fe640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:21:18.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.963+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90100abb00 con 0x7f90101430c0 2024-03-25T18:21:18.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.967+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1774336150 0 0) 0x7f8ffc004eb0 con 0x7f901013f710 2024-03-25T18:21:18.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.967+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9010138e20 con 0x7f901013f710 2024-03-25T18:21:18.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.967+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f90040018f0 con 0x7f901013f710 2024-03-25T18:21:18.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.967+0000 7f8ff77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:18.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.967+0000 7f8ff77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:18.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.967+0000 7f8ff77fe640 10 monclient(hunting): dump: 2024-03-25T18:21:18.969 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:18.969 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:18.969 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:18.969 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:18.969 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:18.969 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:18.969 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:18.969 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:18.969 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:18.969 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:18.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.967+0000 7f8ff77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:18.976 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.975+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2688873647 0 0) 0x7f90100abde0 con 0x7f90100a98f0 2024-03-25T18:21:18.976 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.975+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ffc004eb0 con 0x7f90100a98f0 2024-03-25T18:21:18.976 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.975+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f90000019b0 con 0x7f90100a98f0 2024-03-25T18:21:18.976 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.975+0000 7f8ff77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:18.977 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.975+0000 7f8ff77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:18.977 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.975+0000 7f8ff77fe640 10 monclient(hunting): dump: 2024-03-25T18:21:18.977 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:18.977 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:18.977 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:18.977 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:18.977 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:18.977 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:18.977 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:18.977 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:18.977 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:18.977 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:18.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 813981037 0 0) 0x7f9010138e20 con 0x7f901013f710 2024-03-25T18:21:18.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 >> v1:172.21.15.122:6790/0 conn(0x7f90101430c0 legacy=0x7f90100ac200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 >> v1:172.21.15.122:6789/0 conn(0x7f90100a98f0 legacy=0x7f90101385e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:18.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f8ff77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:21:18.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f8ff77fe640 1 monclient: found mon.b 2024-03-25T18:21:18.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f8ff77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:18.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f8ff77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:18.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9010137b00 con 0x7f901013f710 2024-03-25T18:21:18.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f8ff77fe640 10 monclient: _finish_auth 0 2024-03-25T18:21:18.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f8ff77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:21:18.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f8ff77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:18.984 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f90152f6640 5 monclient: authenticate success, global_id 4447 2024-03-25T18:21:18.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f90152f6640 10 monclient: _renew_subs 2024-03-25T18:21:18.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f90152f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:18.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f90152f6640 1 -- 172.21.15.122:0/4081951894 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90101e34b0 con 0x7f901013f710 2024-03-25T18:21:18.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f90152f6640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:18.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f90152f6640 10 monclient: _renew_subs 2024-03-25T18:21:18.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f90152f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:18.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f90152f6640 1 -- 172.21.15.122:0/4081951894 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f90101e3820 con 0x7f901013f710 2024-03-25T18:21:18.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.983+0000 7f90152f6640 1 librados: init done 2024-03-25T18:21:18.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.984+0000 7f90152f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:18.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.984+0000 7f90152f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:18.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.984+0000 7f90152f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:18.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:18.984+0000 7f90152f6640 1 -- 172.21.15.122:0/4081951894 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9010139060 con 0x7f901013f710 2024-03-25T18:21:19.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.265+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9004001d40 con 0x7f901013f710 2024-03-25T18:21:19.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.265+0000 7f8ff77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:19.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.265+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9004004860 con 0x7f901013f710 2024-03-25T18:21:19.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.265+0000 7f8ff77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:19.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.265+0000 7f8ff77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:19.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.265+0000 7f8ff77fe640 10 monclient: dump: 2024-03-25T18:21:19.266 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:19.266 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:19.267 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:19.267 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:19.267 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:19.267 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:19.267 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:19.267 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:19.267 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:19.267 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:19.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.265+0000 7f900d7fa640 4 set_mon_vals no callback set 2024-03-25T18:21:19.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.266+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f900401a8b0 con 0x7f901013f710 2024-03-25T18:21:19.268 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.266+0000 7f900effd640 10 monclient: get_auth_request con 0x7f8ffc06df70 auth_method 0 2024-03-25T18:21:19.268 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.266+0000 7f8ff77fe640 10 client.4447.objecter ms_handle_connect 0x7f8ffc06df70 2024-03-25T18:21:19.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.271+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f9004054640 con 0x7f901013f710 2024-03-25T18:21:19.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.271+0000 7f8ff77fe640 10 client.4447.objecter ms_dispatch 0x7f9010000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:19.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.271+0000 7f8ff77fe640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:19.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.271+0000 7f8ff77fe640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:19.273 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.271+0000 7f8ff77fe640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-25T18:21:19.273 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.271+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f900404ff80 con 0x7f901013f710 2024-03-25T18:21:19.273 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.271+0000 7f8ff77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:19.273 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.271+0000 7f8ff77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:19.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.521+0000 7f90152f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:19.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.521+0000 7f90152f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:19.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.521+0000 7f90152f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:19.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.521+0000 7f90152f6640 1 -- 172.21.15.122:0/4081951894 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90100627b0 con 0x7f901013f710 2024-03-25T18:21:19.549 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.548+0000 7f8ff77fe640 1 -- 172.21.15.122:0/4081951894 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f9004052080 con 0x7f901013f710 2024-03-25T18:21:19.549 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.548+0000 7f8ff77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:19.549 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.548+0000 7f8ff77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:19.550 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:21:19.550 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:19.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.551+0000 7f90152f6640 10 librados: watch_flush enter 2024-03-25T18:21:19.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.551+0000 7f90152f6640 10 librados: watch_flush exit 2024-03-25T18:21:19.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.551+0000 7f90152f6640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-25T18:21:19.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.551+0000 7f90152f6640 10 client.4447.objecter successfully canceled tick 2024-03-25T18:21:19.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.551+0000 7f90152f6640 1 -- 172.21.15.122:0/4081951894 >> v1:172.21.15.204:6800/676228005 conn(0x7f8ffc06df70 legacy=0x7f8ffc070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:19.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.551+0000 7f90152f6640 10 monclient: shutdown 2024-03-25T18:21:19.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.551+0000 7f90152f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:19.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.551+0000 7f90152f6640 1 -- 172.21.15.122:0/4081951894 >> v1:172.21.15.204:6789/0 conn(0x7f901013f710 legacy=0x7f9010136330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:19.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.551+0000 7f90152f6640 1 -- 172.21.15.122:0/4081951894 shutdown_connections 2024-03-25T18:21:19.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.551+0000 7f90152f6640 1 -- 172.21.15.122:0/4081951894 >> 172.21.15.122:0/4081951894 conn(0x7f90100b7ea0 msgr2=0x7f901014ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:19.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.551+0000 7f90152f6640 1 -- 172.21.15.122:0/4081951894 shutdown_connections 2024-03-25T18:21:19.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.551+0000 7f90152f6640 1 -- 172.21.15.122:0/4081951894 wait complete. 2024-03-25T18:21:19.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:19.551+0000 7f90152f6640 1 librados: shutdown 2024-03-25T18:21:19.563 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:26.342 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:05:20.084 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:26.346 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:05:20.089 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:26.369 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:26.396 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:05:20.141 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:26.396 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:05:20.139 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:26.406 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:05:20.152 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:26.408 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:05:20.150 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:26.415 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:05:20.158 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:26.430 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:05:20.176 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:26.447 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:05:20.189 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:26.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.476+0000 7f6d202d6640 10 monclient: get_monmap_and_config 2024-03-25T18:21:26.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.476+0000 7f6d202d6640 10 monclient: build_initial_monmap 2024-03-25T18:21:26.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.476+0000 7f6d202d6640 10 monclient: monmap: 2024-03-25T18:21:26.477 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:21:26.477 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:26.477 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:21:26.477195+0000 2024-03-25T18:21:26.477 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:21:26.477195+0000 2024-03-25T18:21:26.477 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:26.477 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:26.477 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:21:26.477 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:21:26.477 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:21:26.478 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:26.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.476+0000 7f6d202d6640 1 Processor -- start 2024-03-25T18:21:26.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.476+0000 7f6d202d6640 1 -- start start 2024-03-25T18:21:26.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.476+0000 7f6d202d6640 10 monclient: init 2024-03-25T18:21:26.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.477+0000 7f6d202d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:26.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.477+0000 7f6d202d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:26.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.477+0000 7f6d202d6640 10 monclient(hunting): picked mon.noname-b con 0x7f6d180b4d60 addr v1:172.21.15.122:6790/0 2024-03-25T18:21:26.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.477+0000 7f6d202d6640 10 monclient(hunting): picked mon.noname-a con 0x7f6d180b1730 addr v1:172.21.15.122:6789/0 2024-03-25T18:21:26.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.477+0000 7f6d202d6640 10 monclient(hunting): picked mon.noname-c con 0x7f6d1814be90 addr v1:172.21.15.204:6789/0 2024-03-25T18:21:26.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.477+0000 7f6d202d6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d180af940 con 0x7f6d180b1730 2024-03-25T18:21:26.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.477+0000 7f6d202d6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d180b0a70 con 0x7f6d1814be90 2024-03-25T18:21:26.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.477+0000 7f6d202d6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d180b0d50 con 0x7f6d180b4d60 2024-03-25T18:21:26.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.477+0000 7f6d202d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:26.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.477+0000 7f6d202d6640 10 monclient(hunting): authenticate will time out at 7609.743563s 2024-03-25T18:21:26.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.513+0000 7f6d1e072640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f6d180b4d60 0x7f6d1813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:51308/0 (socket says 172.21.15.122:51308) 2024-03-25T18:21:26.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.513+0000 7f6d1e072640 1 -- 172.21.15.122:0/3361735761 learned_addr learned my addr 172.21.15.122:0/3361735761 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:21:26.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.532+0000 7f6d1d070640 1 -- 172.21.15.122:0/3361735761 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468504233 0 0) 0x7f6d180b0d50 con 0x7f6d180b4d60 2024-03-25T18:21:26.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.532+0000 7f6d1d070640 10 monclient(hunting): _negotiate 2024-03-25T18:21:26.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.532+0000 7f6d1d070640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:26.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.532+0000 7f6d1d070640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:26.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.532+0000 7f6d1d070640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:21:26.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.532+0000 7f6d1d070640 1 -- 172.21.15.122:0/3361735761 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d04003520 con 0x7f6d180b4d60 2024-03-25T18:21:26.545 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.544+0000 7f6d1d070640 1 -- 172.21.15.122:0/3361735761 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 578669039 0 0) 0x7f6d04003520 con 0x7f6d180b4d60 2024-03-25T18:21:26.545 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.544+0000 7f6d1d070640 1 -- 172.21.15.122:0/3361735761 >> v1:172.21.15.204:6789/0 conn(0x7f6d1814be90 legacy=0x7f6d1814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:26.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.544+0000 7f6d1d070640 1 -- 172.21.15.122:0/3361735761 >> v1:172.21.15.122:6789/0 conn(0x7f6d180b1730 legacy=0x7f6d180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:26.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.544+0000 7f6d1d070640 10 monclient: _finish_hunting 0 2024-03-25T18:21:26.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.544+0000 7f6d1d070640 1 monclient: found mon.noname-b 2024-03-25T18:21:26.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.544+0000 7f6d1d070640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:26.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.544+0000 7f6d1d070640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:21:26.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.544+0000 7f6d1d070640 1 -- 172.21.15.122:0/3361735761 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d180b1030 con 0x7f6d180b4d60 2024-03-25T18:21:26.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.544+0000 7f6d1d070640 10 monclient: _finish_auth 0 2024-03-25T18:21:26.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.544+0000 7f6d1d070640 10 monclient: _check_auth_tickets 2024-03-25T18:21:26.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.544+0000 7f6d1d070640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:26.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.544+0000 7f6d1d070640 1 -- 172.21.15.122:0/3361735761 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f6d14001690 con 0x7f6d180b4d60 2024-03-25T18:21:26.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.544+0000 7f6d1d070640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:26.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.544+0000 7f6d1d070640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:26.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.544+0000 7f6d1d070640 10 monclient: dump: 2024-03-25T18:21:26.546 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:26.546 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:26.546 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:26.547 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:26.547 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:26.547 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:26.547 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:26.547 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:26.547 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:26.547 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:26.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.544+0000 7f6d202d6640 5 monclient: authenticate success, global_id 4427 2024-03-25T18:21:26.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.544+0000 7f6d202d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:26.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.570+0000 7f6d1d070640 1 -- 172.21.15.122:0/3361735761 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d140017f0 con 0x7f6d180b4d60 2024-03-25T18:21:26.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.570+0000 7f6d1d070640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:26.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.570+0000 7f6d1d070640 1 -- 172.21.15.122:0/3361735761 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f6d14001c00 con 0x7f6d180b4d60 2024-03-25T18:21:26.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.570+0000 7f6d202d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:26.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.570+0000 7f6d202d6640 4 set_mon_vals no callback set 2024-03-25T18:21:26.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.570+0000 7f6d1d070640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:26.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.570+0000 7f6d1d070640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:26.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.570+0000 7f6d1d070640 10 monclient: dump: 2024-03-25T18:21:26.571 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:26.571 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:26.571 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:26.571 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:26.572 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:26.572 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:26.572 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:26.572 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:26.572 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:26.572 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:26.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.571+0000 7f6d202d6640 10 monclient: shutdown 2024-03-25T18:21:26.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.571+0000 7f6d202d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:26.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.571+0000 7f6d202d6640 1 -- 172.21.15.122:0/3361735761 >> v1:172.21.15.122:6790/0 conn(0x7f6d180b4d60 legacy=0x7f6d1813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:26.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.571+0000 7f6d202d6640 1 -- 172.21.15.122:0/3361735761 shutdown_connections 2024-03-25T18:21:26.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.571+0000 7f6d202d6640 1 -- 172.21.15.122:0/3361735761 >> 172.21.15.122:0/3361735761 conn(0x7f6d180617a0 msgr2=0x7f6d18061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:26.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.571+0000 7f6d202d6640 1 -- 172.21.15.122:0/3361735761 shutdown_connections 2024-03-25T18:21:26.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.571+0000 7f6d202d6640 1 -- 172.21.15.122:0/3361735761 wait complete. 2024-03-25T18:21:26.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.571+0000 7f6d202d6640 10 monclient: build_initial_monmap 2024-03-25T18:21:26.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.571+0000 7f6d202d6640 10 monclient: monmap: 2024-03-25T18:21:26.574 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:21:26.574 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:26.574 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:21:26.574 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:21:26.574 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:26.574 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:21:26.574 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:21:26.574 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:21:26.574 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:21:26.574 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:26.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.572+0000 7f6d202d6640 1 librados: starting msgr at 2024-03-25T18:21:26.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.572+0000 7f6d202d6640 1 librados: starting objecter 2024-03-25T18:21:26.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.572+0000 7f6d202d6640 1 Processor -- start 2024-03-25T18:21:26.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.572+0000 7f6d202d6640 1 -- start start 2024-03-25T18:21:26.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.572+0000 7f6d202d6640 1 librados: setting wanted keys 2024-03-25T18:21:26.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.572+0000 7f6d202d6640 1 librados: calling monclient init 2024-03-25T18:21:26.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.572+0000 7f6d202d6640 10 monclient: init 2024-03-25T18:21:26.576 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.573+0000 7f6d202d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:26.576 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.573+0000 7f6d202d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:26.576 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.573+0000 7f6d202d6640 10 monclient(hunting): picked mon.noname-b con 0x7f6d180b1730 addr v1:172.21.15.204:6789/0 2024-03-25T18:21:26.576 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.573+0000 7f6d202d6640 10 monclient(hunting): picked mon.noname-a con 0x7f6d180b4d60 addr v1:172.21.15.122:6789/0 2024-03-25T18:21:26.576 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.573+0000 7f6d202d6640 10 monclient(hunting): picked mon.noname-c con 0x7f6d1814be90 addr v1:172.21.15.122:6790/0 2024-03-25T18:21:26.576 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.573+0000 7f6d202d6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d181361e0 con 0x7f6d180b4d60 2024-03-25T18:21:26.576 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.573+0000 7f6d202d6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d18145080 con 0x7f6d180b1730 2024-03-25T18:21:26.576 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.573+0000 7f6d202d6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d18145210 con 0x7f6d1814be90 2024-03-25T18:21:26.576 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.573+0000 7f6d202d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:26.576 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.573+0000 7f6d202d6640 10 monclient(hunting): authenticate will time out at 7609.839067s 2024-03-25T18:21:26.583 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.582+0000 7f6d1e072640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f6d180b1730 0x7f6d1812ff90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:37958/0 (socket says 172.21.15.122:37958) 2024-03-25T18:21:26.583 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.582+0000 7f6d1e072640 1 -- 172.21.15.122:0/2773127857 learned_addr learned my addr 172.21.15.122:0/2773127857 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:21:26.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.583+0000 7f6d0effd640 10 client.?.objecter ms_handle_connect 0x7f6d180b1730 2024-03-25T18:21:26.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.583+0000 7f6d0effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:26.590 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.589+0000 7f6d0effd640 10 client.?.objecter ms_handle_connect 0x7f6d180b4d60 2024-03-25T18:21:26.590 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.589+0000 7f6d0effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:26.591 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.590+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856910992 0 0) 0x7f6d18145080 con 0x7f6d180b1730 2024-03-25T18:21:26.591 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.590+0000 7f6d0effd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:26.591 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.590+0000 7f6d0effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:26.591 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.590+0000 7f6d0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:26.591 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.590+0000 7f6d0effd640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:21:26.591 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.590+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cfc004eb0 con 0x7f6d180b1730 2024-03-25T18:21:26.598 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.597+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1602997798 0 0) 0x7f6d181361e0 con 0x7f6d180b4d60 2024-03-25T18:21:26.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.597+0000 7f6d0effd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:26.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.597+0000 7f6d0effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:26.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.597+0000 7f6d0effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:26.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.597+0000 7f6d0effd640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:21:26.599 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.597+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d18145080 con 0x7f6d180b4d60 2024-03-25T18:21:26.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.605+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1858687599 0 0) 0x7f6d18145080 con 0x7f6d180b4d60 2024-03-25T18:21:26.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.605+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d181361e0 con 0x7f6d180b4d60 2024-03-25T18:21:26.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.605+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f6d08001860 con 0x7f6d180b4d60 2024-03-25T18:21:26.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.605+0000 7f6d0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:26.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.605+0000 7f6d0effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:26.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.605+0000 7f6d0effd640 10 monclient(hunting): dump: 2024-03-25T18:21:26.609 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:26.609 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:26.609 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:26.609 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:26.609 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:26.609 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:26.610 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:26.610 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:26.610 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:26.610 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:26.610 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.605+0000 7f6d0effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:26.613 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.611+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 952552183 0 0) 0x7f6cfc004eb0 con 0x7f6d180b1730 2024-03-25T18:21:26.613 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.611+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d18145080 con 0x7f6d180b1730 2024-03-25T18:21:26.613 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.611+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f6d14001c00 con 0x7f6d180b1730 2024-03-25T18:21:26.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.611+0000 7f6d0effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:26.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.612+0000 7f6d0effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:26.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.612+0000 7f6d0effd640 10 monclient(hunting): dump: 2024-03-25T18:21:26.614 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:26.614 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:26.614 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:26.614 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:26.614 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:26.614 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:26.614 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:26.614 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:26.614 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:26.614 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:26.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.613+0000 7f6d0effd640 10 client.?.objecter ms_handle_connect 0x7f6d1814be90 2024-03-25T18:21:26.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.613+0000 7f6d0effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:26.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3846989026 0 0) 0x7f6d181361e0 con 0x7f6d180b4d60 2024-03-25T18:21:26.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 >> v1:172.21.15.122:6790/0 conn(0x7f6d1814be90 legacy=0x7f6d181e2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:26.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 >> v1:172.21.15.204:6789/0 conn(0x7f6d180b1730 legacy=0x7f6d1812ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:26.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d0effd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:26.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d0effd640 1 monclient: found mon.a 2024-03-25T18:21:26.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d0effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:26.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d0effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:21:26.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d181306a0 con 0x7f6d180b4d60 2024-03-25T18:21:26.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d0effd640 10 monclient: _finish_auth 0 2024-03-25T18:21:26.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d0effd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:26.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d0effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:26.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d202d6640 5 monclient: authenticate success, global_id 4458 2024-03-25T18:21:26.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d202d6640 10 monclient: _renew_subs 2024-03-25T18:21:26.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d202d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:21:26.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d202d6640 1 -- 172.21.15.122:0/2773127857 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d18145440 con 0x7f6d180b4d60 2024-03-25T18:21:26.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d202d6640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:26.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d202d6640 10 monclient: _renew_subs 2024-03-25T18:21:26.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d202d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:21:26.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d202d6640 1 -- 172.21.15.122:0/2773127857 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d18137ea0 con 0x7f6d180b4d60 2024-03-25T18:21:26.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.617+0000 7f6d202d6640 1 librados: init done 2024-03-25T18:21:26.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.618+0000 7f6d202d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:26.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.618+0000 7f6d202d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:26.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.618+0000 7f6d202d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:21:26.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.618+0000 7f6d202d6640 1 -- 172.21.15.122:0/2773127857 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d18130890 con 0x7f6d180b4d60 2024-03-25T18:21:26.623 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.622+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d08001d10 con 0x7f6d180b4d60 2024-03-25T18:21:26.623 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.622+0000 7f6d0effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:26.623 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.622+0000 7f6d1c86f640 4 set_mon_vals no callback set 2024-03-25T18:21:26.921 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.920+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f6d080037b0 con 0x7f6d180b4d60 2024-03-25T18:21:26.922 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.920+0000 7f6d0effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:26.922 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.920+0000 7f6d0effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:26.922 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.920+0000 7f6d0effd640 10 monclient: dump: 2024-03-25T18:21:26.922 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:26.922 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:26.922 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:26.922 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:26.922 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:26.922 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:26.922 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:26.922 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:26.922 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:26.922 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:26.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.921+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f6d08003910 con 0x7f6d180b4d60 2024-03-25T18:21:26.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.922+0000 7f6d1e072640 10 monclient: get_auth_request con 0x7f6cfc06de30 auth_method 0 2024-03-25T18:21:26.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.922+0000 7f6d0effd640 10 client.4458.objecter ms_handle_connect 0x7f6cfc06de30 2024-03-25T18:21:26.929 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.927+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f6d080535f0 con 0x7f6d180b4d60 2024-03-25T18:21:26.929 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.927+0000 7f6d0effd640 10 client.4458.objecter ms_dispatch 0x7f6d18000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:26.929 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.927+0000 7f6d0effd640 3 client.4458.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:26.929 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.927+0000 7f6d0effd640 3 client.4458.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:26.929 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.927+0000 7f6d0effd640 20 client.4458.objecter dump_active .. 0 homeless 2024-03-25T18:21:26.929 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.927+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d0804ef30 con 0x7f6d180b4d60 2024-03-25T18:21:26.929 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.928+0000 7f6d0effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:26.929 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:26.928+0000 7f6d0effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:27.178 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.177+0000 7f6d202d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:27.178 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.177+0000 7f6d202d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:27.178 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.177+0000 7f6d202d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:21:27.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.177+0000 7f6d202d6640 1 -- 172.21.15.122:0/2773127857 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d181455d0 con 0x7f6d180b4d60 2024-03-25T18:21:27.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.198+0000 7f6d0effd640 1 -- 172.21.15.122:0/2773127857 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f6d08051030 con 0x7f6d180b4d60 2024-03-25T18:21:27.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.198+0000 7f6d0effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:27.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.198+0000 7f6d0effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:27.199 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:21:27.199 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:27.202 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.200+0000 7f6d202d6640 10 librados: watch_flush enter 2024-03-25T18:21:27.202 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.200+0000 7f6d202d6640 10 librados: watch_flush exit 2024-03-25T18:21:27.202 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.200+0000 7f6d202d6640 20 client.4458.objecter shutdown clearing up homeless session... 2024-03-25T18:21:27.203 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.200+0000 7f6d202d6640 10 client.4458.objecter successfully canceled tick 2024-03-25T18:21:27.203 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.201+0000 7f6d202d6640 1 -- 172.21.15.122:0/2773127857 >> v1:172.21.15.204:6800/676228005 conn(0x7f6cfc06de30 legacy=0x7f6cfc0702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:27.203 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.201+0000 7f6d202d6640 10 monclient: shutdown 2024-03-25T18:21:27.203 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.201+0000 7f6d202d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:27.203 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.201+0000 7f6d202d6640 1 -- 172.21.15.122:0/2773127857 >> v1:172.21.15.122:6789/0 conn(0x7f6d180b4d60 legacy=0x7f6d181d76c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:27.203 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.201+0000 7f6d202d6640 1 -- 172.21.15.122:0/2773127857 shutdown_connections 2024-03-25T18:21:27.203 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.201+0000 7f6d202d6640 1 -- 172.21.15.122:0/2773127857 >> 172.21.15.122:0/2773127857 conn(0x7f6d180617a0 msgr2=0x7f6d180b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:27.203 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.201+0000 7f6d202d6640 1 -- 172.21.15.122:0/2773127857 shutdown_connections 2024-03-25T18:21:27.203 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.201+0000 7f6d202d6640 1 -- 172.21.15.122:0/2773127857 wait complete. 2024-03-25T18:21:27.203 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:27.201+0000 7f6d202d6640 1 librados: shutdown 2024-03-25T18:21:27.214 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:34.020 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:34.134 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.133+0000 7f54abf9d640 10 monclient: get_monmap_and_config 2024-03-25T18:21:34.147 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.133+0000 7f54abf9d640 10 monclient: build_initial_monmap 2024-03-25T18:21:34.147 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.133+0000 7f54abf9d640 10 monclient: monmap: 2024-03-25T18:21:34.147 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:21:34.147 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:34.147 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:21:34.134652+0000 2024-03-25T18:21:34.147 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:21:34.134652+0000 2024-03-25T18:21:34.147 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:34.147 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:34.147 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:21:34.147 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:21:34.148 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:21:34.148 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:34.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.134+0000 7f54abf9d640 1 Processor -- start 2024-03-25T18:21:34.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.134+0000 7f54abf9d640 1 -- start start 2024-03-25T18:21:34.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.134+0000 7f54abf9d640 10 monclient: init 2024-03-25T18:21:34.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.135+0000 7f54abf9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:34.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.135+0000 7f54abf9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:34.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.135+0000 7f54abf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f54a40a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:21:34.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.135+0000 7f54abf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f54a413f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:21:34.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.135+0000 7f54abf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f54a41430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:21:34.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.135+0000 7f54abf9d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a4142520 con 0x7f54a413f710 2024-03-25T18:21:34.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.135+0000 7f54abf9d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a40b28a0 con 0x7f54a40a98f0 2024-03-25T18:21:34.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.135+0000 7f54abf9d640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a40b2a30 con 0x7f54a41430c0 2024-03-25T18:21:34.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.135+0000 7f54abf9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:34.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.135+0000 7f54abf9d640 10 monclient(hunting): authenticate will time out at 7617.401888s 2024-03-25T18:21:34.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.147+0000 7f54aa53a640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f54a41430c0 0x7f54a414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:44944/0 (socket says 172.21.15.122:44944) 2024-03-25T18:21:34.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.147+0000 7f54aa53a640 1 -- 172.21.15.122:0/1137078346 learned_addr learned my addr 172.21.15.122:0/1137078346 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:21:34.153 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.152+0000 7f54a8d37640 1 -- 172.21.15.122:0/1137078346 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3487377092 0 0) 0x7f54a4142520 con 0x7f54a413f710 2024-03-25T18:21:34.153 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.152+0000 7f54a8d37640 10 monclient(hunting): _negotiate 2024-03-25T18:21:34.153 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.152+0000 7f54a8d37640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:34.153 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.152+0000 7f54a8d37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:34.153 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.152+0000 7f54a8d37640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:21:34.153 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.152+0000 7f54a8d37640 1 -- 172.21.15.122:0/1137078346 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5490003520 con 0x7f54a413f710 2024-03-25T18:21:34.169 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.167+0000 7f54a8d37640 1 -- 172.21.15.122:0/1137078346 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1723356039 0 0) 0x7f5490003520 con 0x7f54a413f710 2024-03-25T18:21:34.169 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.167+0000 7f54a8d37640 1 -- 172.21.15.122:0/1137078346 >> v1:172.21.15.122:6790/0 conn(0x7f54a41430c0 legacy=0x7f54a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:34.169 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.167+0000 7f54a8d37640 1 -- 172.21.15.122:0/1137078346 >> v1:172.21.15.204:6789/0 conn(0x7f54a40a98f0 legacy=0x7f54a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:34.169 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.167+0000 7f54a8d37640 10 monclient: _finish_hunting 0 2024-03-25T18:21:34.169 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.167+0000 7f54a8d37640 1 monclient: found mon.noname-a 2024-03-25T18:21:34.169 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.167+0000 7f54a8d37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:34.169 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.167+0000 7f54a8d37640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:21:34.169 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.167+0000 7f54a8d37640 1 -- 172.21.15.122:0/1137078346 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a40b44c0 con 0x7f54a413f710 2024-03-25T18:21:34.169 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.167+0000 7f54a8d37640 10 monclient: _finish_auth 0 2024-03-25T18:21:34.169 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.167+0000 7f54a8d37640 10 monclient: _check_auth_tickets 2024-03-25T18:21:34.169 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.167+0000 7f54a8d37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:34.170 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.167+0000 7f54a8d37640 1 -- 172.21.15.122:0/1137078346 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5494001690 con 0x7f54a413f710 2024-03-25T18:21:34.170 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.167+0000 7f54a8d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:34.170 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.167+0000 7f54a8d37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:21:34.170 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.167+0000 7f54a8d37640 10 monclient: dump: 2024-03-25T18:21:34.170 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:34.170 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:34.170 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:34.170 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:34.170 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:34.170 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:34.170 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:34.170 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:34.170 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:34.171 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:34.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.167+0000 7f54abf9d640 5 monclient: authenticate success, global_id 4461 2024-03-25T18:21:34.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.167+0000 7f54abf9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:34.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.188+0000 7f54a8d37640 1 -- 172.21.15.122:0/1137078346 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54940017f0 con 0x7f54a413f710 2024-03-25T18:21:34.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.188+0000 7f54a8d37640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:34.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.188+0000 7f54a8d37640 1 -- 172.21.15.122:0/1137078346 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5494001c00 con 0x7f54a413f710 2024-03-25T18:21:34.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.188+0000 7f54abf9d640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:34.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.188+0000 7f54abf9d640 4 set_mon_vals no callback set 2024-03-25T18:21:34.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.188+0000 7f54a8d37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:34.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.188+0000 7f54a8d37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:34.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.188+0000 7f54a8d37640 10 monclient: dump: 2024-03-25T18:21:34.190 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:34.190 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:34.190 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:34.190 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:34.190 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:34.190 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:34.190 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:34.190 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:34.190 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:34.190 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:34.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.189+0000 7f54abf9d640 10 monclient: shutdown 2024-03-25T18:21:34.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.189+0000 7f54abf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:34.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.189+0000 7f54abf9d640 1 -- 172.21.15.122:0/1137078346 >> v1:172.21.15.122:6789/0 conn(0x7f54a413f710 legacy=0x7f54a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:34.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.189+0000 7f54abf9d640 1 -- 172.21.15.122:0/1137078346 shutdown_connections 2024-03-25T18:21:34.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.189+0000 7f54abf9d640 1 -- 172.21.15.122:0/1137078346 >> 172.21.15.122:0/1137078346 conn(0x7f54a40b7ea0 msgr2=0x7f54a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:34.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.189+0000 7f54abf9d640 1 -- 172.21.15.122:0/1137078346 shutdown_connections 2024-03-25T18:21:34.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.189+0000 7f54abf9d640 1 -- 172.21.15.122:0/1137078346 wait complete. 2024-03-25T18:21:34.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.190+0000 7f54abf9d640 10 monclient: build_initial_monmap 2024-03-25T18:21:34.193 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.190+0000 7f54abf9d640 10 monclient: monmap: 2024-03-25T18:21:34.193 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:21:34.193 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:34.193 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:21:34.193 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:21:34.193 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:34.193 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:21:34.194 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:21:34.194 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:21:34.194 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:21:34.194 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:34.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.190+0000 7f54abf9d640 1 librados: starting msgr at 2024-03-25T18:21:34.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.190+0000 7f54abf9d640 1 librados: starting objecter 2024-03-25T18:21:34.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.190+0000 7f54abf9d640 1 Processor -- start 2024-03-25T18:21:34.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.190+0000 7f54abf9d640 1 -- start start 2024-03-25T18:21:34.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.190+0000 7f54abf9d640 1 librados: setting wanted keys 2024-03-25T18:21:34.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.190+0000 7f54abf9d640 1 librados: calling monclient init 2024-03-25T18:21:34.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.190+0000 7f54abf9d640 10 monclient: init 2024-03-25T18:21:34.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.191+0000 7f54abf9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:34.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.191+0000 7f54abf9d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:34.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.191+0000 7f54abf9d640 10 monclient(hunting): picked mon.noname-c con 0x7f54a40a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:21:34.194 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.191+0000 7f54abf9d640 10 monclient(hunting): picked mon.noname-a con 0x7f54a413f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:21:34.195 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.191+0000 7f54abf9d640 10 monclient(hunting): picked mon.noname-b con 0x7f54a41430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:21:34.195 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.191+0000 7f54abf9d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a4139a80 con 0x7f54a413f710 2024-03-25T18:21:34.195 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.191+0000 7f54abf9d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a4137450 con 0x7f54a41430c0 2024-03-25T18:21:34.195 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.191+0000 7f54abf9d640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54a41d7ba0 con 0x7f54a40a98f0 2024-03-25T18:21:34.195 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.191+0000 7f54abf9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:34.195 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.191+0000 7f54abf9d640 10 monclient(hunting): authenticate will time out at 7617.457638s 2024-03-25T18:21:34.195 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.193+0000 7f54a9538640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f54a413f710 0x7f54a41d7490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58964/0 (socket says 172.21.15.122:58964) 2024-03-25T18:21:34.196 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.193+0000 7f54a9538640 1 -- 172.21.15.122:0/455908345 learned_addr learned my addr 172.21.15.122:0/455908345 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:21:34.196 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.194+0000 7f549a7fc640 10 client.?.objecter ms_handle_connect 0x7f54a413f710 2024-03-25T18:21:34.196 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.194+0000 7f549a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:34.210 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.209+0000 7f549a7fc640 10 client.?.objecter ms_handle_connect 0x7f54a41430c0 2024-03-25T18:21:34.211 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.209+0000 7f549a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:34.212 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.211+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 643618198 0 0) 0x7f54a40b44c0 con 0x7f54a413f710 2024-03-25T18:21:34.212 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.211+0000 7f549a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:34.212 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.211+0000 7f549a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:34.212 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.211+0000 7f549a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:34.212 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.211+0000 7f549a7fc640 10 monclient(hunting): my global_id is 4464 2024-03-25T18:21:34.212 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.211+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5488004eb0 con 0x7f54a413f710 2024-03-25T18:21:34.213 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.212+0000 7f549a7fc640 10 client.?.objecter ms_handle_connect 0x7f54a40a98f0 2024-03-25T18:21:34.213 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.212+0000 7f549a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:34.218 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.217+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342318467 0 0) 0x7f54a4137450 con 0x7f54a41430c0 2024-03-25T18:21:34.219 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.217+0000 7f549a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:34.219 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.217+0000 7f549a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:34.219 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.217+0000 7f549a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:34.219 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.217+0000 7f549a7fc640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:21:34.219 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.217+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a40b44c0 con 0x7f54a41430c0 2024-03-25T18:21:34.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.224+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798572569 0 0) 0x7f54a41d7ba0 con 0x7f54a40a98f0 2024-03-25T18:21:34.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.224+0000 7f549a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:34.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.224+0000 7f549a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:34.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.224+0000 7f549a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:34.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.224+0000 7f549a7fc640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:21:34.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.224+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54a4137450 con 0x7f54a40a98f0 2024-03-25T18:21:34.229 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.227+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1221696874 0 0) 0x7f54a40b44c0 con 0x7f54a41430c0 2024-03-25T18:21:34.229 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.227+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54a41d7ba0 con 0x7f54a41430c0 2024-03-25T18:21:34.229 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.227+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f549c001b10 con 0x7f54a41430c0 2024-03-25T18:21:34.229 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.227+0000 7f549a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:34.229 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.227+0000 7f549a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:34.229 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.227+0000 7f549a7fc640 10 monclient(hunting): dump: 2024-03-25T18:21:34.229 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:34.229 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:34.229 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:34.229 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:34.229 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:34.229 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:34.229 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:34.230 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:34.230 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:34.230 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:34.230 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.227+0000 7f549a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:34.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.236+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2951308751 0 0) 0x7f54a4137450 con 0x7f54a40a98f0 2024-03-25T18:21:34.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.236+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54a40b44c0 con 0x7f54a40a98f0 2024-03-25T18:21:34.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.236+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f54a0001830 con 0x7f54a40a98f0 2024-03-25T18:21:34.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.236+0000 7f549a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:34.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.236+0000 7f549a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:34.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.236+0000 7f549a7fc640 10 monclient(hunting): dump: 2024-03-25T18:21:34.237 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:34.237 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:34.237 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:34.238 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:34.238 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:34.238 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:34.238 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:34.238 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:34.238 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:34.238 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:34.239 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1321460427 0 0) 0x7f54a41d7ba0 con 0x7f54a41430c0 2024-03-25T18:21:34.239 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 >> v1:172.21.15.122:6790/0 conn(0x7f54a40a98f0 legacy=0x7f54a4139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:34.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 >> v1:172.21.15.122:6789/0 conn(0x7f54a413f710 legacy=0x7f54a41d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:34.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f549a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:34.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f549a7fc640 1 monclient: found mon.b 2024-03-25T18:21:34.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f549a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:34.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f549a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:34.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54a41368f0 con 0x7f54a41430c0 2024-03-25T18:21:34.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f549a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:34.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f549a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:34.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f549a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:34.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f54abf9d640 5 monclient: authenticate success, global_id 4459 2024-03-25T18:21:34.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f54abf9d640 10 monclient: _renew_subs 2024-03-25T18:21:34.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f54abf9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:34.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f54abf9d640 1 -- 172.21.15.122:0/455908345 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a4136b40 con 0x7f54a41430c0 2024-03-25T18:21:34.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f54abf9d640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:34.241 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f54abf9d640 10 monclient: _renew_subs 2024-03-25T18:21:34.241 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f54abf9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:34.241 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f54abf9d640 1 -- 172.21.15.122:0/455908345 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a4137ba0 con 0x7f54a41430c0 2024-03-25T18:21:34.241 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.237+0000 7f54abf9d640 1 librados: init done 2024-03-25T18:21:34.241 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.238+0000 7f54abf9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:34.241 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.238+0000 7f54abf9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:34.241 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.238+0000 7f54abf9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:34.241 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.238+0000 7f54abf9d640 1 -- 172.21.15.122:0/455908345 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a40627b0 con 0x7f54a41430c0 2024-03-25T18:21:34.529 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.527+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f549c001f40 con 0x7f54a41430c0 2024-03-25T18:21:34.529 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.527+0000 7f549a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:34.529 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.527+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f549c004b60 con 0x7f54a41430c0 2024-03-25T18:21:34.529 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.527+0000 7f549a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:34.529 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.527+0000 7f549a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:34.529 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.527+0000 7f549a7fc640 10 monclient: dump: 2024-03-25T18:21:34.529 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:34.529 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:34.529 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:34.530 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:34.530 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:34.530 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:34.530 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:34.530 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:34.530 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:34.530 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:34.530 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.527+0000 7f549bfff640 4 set_mon_vals no callback set 2024-03-25T18:21:34.531 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.528+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f549c004d80 con 0x7f54a41430c0 2024-03-25T18:21:34.531 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.529+0000 7f54a9d39640 10 monclient: get_auth_request con 0x7f548806df80 auth_method 0 2024-03-25T18:21:34.531 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.529+0000 7f549a7fc640 10 client.4459.objecter ms_handle_connect 0x7f548806df80 2024-03-25T18:21:34.535 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.534+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f549c054820 con 0x7f54a41430c0 2024-03-25T18:21:34.535 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.534+0000 7f549a7fc640 10 client.4459.objecter ms_dispatch 0x7f54a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:34.535 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.534+0000 7f549a7fc640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:34.535 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.534+0000 7f549a7fc640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:34.535 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.534+0000 7f549a7fc640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-25T18:21:34.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.535+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f549c050330 con 0x7f54a41430c0 2024-03-25T18:21:34.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.535+0000 7f549a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:34.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.535+0000 7f549a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:34.778 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.777+0000 7f54abf9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:34.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.777+0000 7f54abf9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:34.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.777+0000 7f54abf9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:34.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.777+0000 7f54abf9d640 1 -- 172.21.15.122:0/455908345 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54a4074cc0 con 0x7f54a41430c0 2024-03-25T18:21:34.863 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.861+0000 7f549a7fc640 1 -- 172.21.15.122:0/455908345 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f549c052260 con 0x7f54a41430c0 2024-03-25T18:21:34.863 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.862+0000 7f549a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:34.863 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.862+0000 7f549a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:34.863 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:21:34.863 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:34.865 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.864+0000 7f54abf9d640 10 librados: watch_flush enter 2024-03-25T18:21:34.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.864+0000 7f54abf9d640 10 librados: watch_flush exit 2024-03-25T18:21:34.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.864+0000 7f54abf9d640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-25T18:21:34.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.864+0000 7f54abf9d640 10 client.4459.objecter successfully canceled tick 2024-03-25T18:21:34.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.864+0000 7f54abf9d640 1 -- 172.21.15.122:0/455908345 >> v1:172.21.15.204:6800/676228005 conn(0x7f548806df80 legacy=0x7f5488070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:34.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.864+0000 7f54abf9d640 10 monclient: shutdown 2024-03-25T18:21:34.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.864+0000 7f54abf9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:34.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.864+0000 7f54abf9d640 1 -- 172.21.15.122:0/455908345 >> v1:172.21.15.204:6789/0 conn(0x7f54a41430c0 legacy=0x7f54a4132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:34.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.864+0000 7f54abf9d640 1 -- 172.21.15.122:0/455908345 shutdown_connections 2024-03-25T18:21:34.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.864+0000 7f54abf9d640 1 -- 172.21.15.122:0/455908345 >> 172.21.15.122:0/455908345 conn(0x7f54a40b7ea0 msgr2=0x7f54a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:34.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.864+0000 7f54abf9d640 1 -- 172.21.15.122:0/455908345 shutdown_connections 2024-03-25T18:21:34.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.864+0000 7f54abf9d640 1 -- 172.21.15.122:0/455908345 wait complete. 2024-03-25T18:21:34.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:34.864+0000 7f54abf9d640 1 librados: shutdown 2024-03-25T18:21:34.877 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:41.683 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:41.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.799+0000 7fe468e63640 10 monclient: get_monmap_and_config 2024-03-25T18:21:41.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.799+0000 7fe468e63640 10 monclient: build_initial_monmap 2024-03-25T18:21:41.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.799+0000 7fe468e63640 10 monclient: monmap: 2024-03-25T18:21:41.800 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:21:41.800 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:41.801 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:21:41.800288+0000 2024-03-25T18:21:41.801 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:21:41.800288+0000 2024-03-25T18:21:41.801 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:41.801 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:41.801 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:21:41.801 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:21:41.801 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:21:41.801 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:41.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.799+0000 7fe468e63640 1 Processor -- start 2024-03-25T18:21:41.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.799+0000 7fe468e63640 1 -- start start 2024-03-25T18:21:41.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.799+0000 7fe468e63640 10 monclient: init 2024-03-25T18:21:41.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.800+0000 7fe468e63640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:41.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.800+0000 7fe468e63640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:41.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.800+0000 7fe468e63640 10 monclient(hunting): picked mon.noname-c con 0x7fe4640a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:21:41.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.800+0000 7fe468e63640 10 monclient(hunting): picked mon.noname-a con 0x7fe46413f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:21:41.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.800+0000 7fe468e63640 10 monclient(hunting): picked mon.noname-b con 0x7fe4641430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:21:41.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.800+0000 7fe468e63640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe464142520 con 0x7fe46413f710 2024-03-25T18:21:41.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.800+0000 7fe468e63640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4640b28a0 con 0x7fe4640a98f0 2024-03-25T18:21:41.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.800+0000 7fe468e63640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4640b2a30 con 0x7fe4641430c0 2024-03-25T18:21:41.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.800+0000 7fe468e63640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:41.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.800+0000 7fe468e63640 10 monclient(hunting): authenticate will time out at 7625.066806s 2024-03-25T18:21:41.815 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.814+0000 7fe462d9d640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7fe4641430c0 0x7fe46414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:48882/0 (socket says 172.21.15.122:48882) 2024-03-25T18:21:41.815 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.814+0000 7fe462d9d640 1 -- 172.21.15.122:0/302781569 learned_addr learned my addr 172.21.15.122:0/302781569 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:21:41.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.826+0000 7fe46159a640 1 -- 172.21.15.122:0/302781569 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582356580 0 0) 0x7fe464142520 con 0x7fe46413f710 2024-03-25T18:21:41.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.826+0000 7fe46159a640 10 monclient(hunting): _negotiate 2024-03-25T18:21:41.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.826+0000 7fe46159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:41.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.826+0000 7fe46159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:41.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.826+0000 7fe46159a640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:21:41.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.826+0000 7fe46159a640 1 -- 172.21.15.122:0/302781569 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe44c003520 con 0x7fe46413f710 2024-03-25T18:21:41.830 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.828+0000 7fe46159a640 1 -- 172.21.15.122:0/302781569 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1054233111 0 0) 0x7fe4640b2a30 con 0x7fe4641430c0 2024-03-25T18:21:41.830 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.828+0000 7fe46159a640 10 monclient(hunting): _negotiate 2024-03-25T18:21:41.830 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.828+0000 7fe46159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:41.830 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.828+0000 7fe46159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:41.830 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.828+0000 7fe46159a640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:21:41.830 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.828+0000 7fe46159a640 1 -- 172.21.15.122:0/302781569 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe464142520 con 0x7fe4641430c0 2024-03-25T18:21:41.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.833+0000 7fe46159a640 1 -- 172.21.15.122:0/302781569 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850435288 0 0) 0x7fe4640b28a0 con 0x7fe4640a98f0 2024-03-25T18:21:41.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.833+0000 7fe46159a640 10 monclient(hunting): _negotiate 2024-03-25T18:21:41.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.833+0000 7fe46159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:41.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.833+0000 7fe46159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:41.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.833+0000 7fe46159a640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:21:41.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.833+0000 7fe46159a640 1 -- 172.21.15.122:0/302781569 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4640b2a30 con 0x7fe4640a98f0 2024-03-25T18:21:41.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.846+0000 7fe46159a640 1 -- 172.21.15.122:0/302781569 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2463699646 0 0) 0x7fe464142520 con 0x7fe4641430c0 2024-03-25T18:21:41.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.846+0000 7fe46159a640 1 -- 172.21.15.122:0/302781569 >> v1:172.21.15.204:6789/0 conn(0x7fe4640a98f0 legacy=0x7fe4640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:41.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.846+0000 7fe46159a640 1 -- 172.21.15.122:0/302781569 >> v1:172.21.15.122:6789/0 conn(0x7fe46413f710 legacy=0x7fe4641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:41.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.846+0000 7fe46159a640 10 monclient: _finish_hunting 0 2024-03-25T18:21:41.848 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.846+0000 7fe46159a640 1 monclient: found mon.noname-b 2024-03-25T18:21:41.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.846+0000 7fe46159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:41.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.846+0000 7fe46159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:21:41.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.846+0000 7fe46159a640 1 -- 172.21.15.122:0/302781569 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4640b44c0 con 0x7fe4641430c0 2024-03-25T18:21:41.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.846+0000 7fe46159a640 10 monclient: _finish_auth 0 2024-03-25T18:21:41.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.846+0000 7fe46159a640 10 monclient: _check_auth_tickets 2024-03-25T18:21:41.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.846+0000 7fe46159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:41.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.846+0000 7fe46159a640 1 -- 172.21.15.122:0/302781569 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fe45c0019f0 con 0x7fe4641430c0 2024-03-25T18:21:41.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.846+0000 7fe468e63640 5 monclient: authenticate success, global_id 4439 2024-03-25T18:21:41.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.846+0000 7fe468e63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:41.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.846+0000 7fe46159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:41.849 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.846+0000 7fe46159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:41.850 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.846+0000 7fe46159a640 10 monclient: dump: 2024-03-25T18:21:41.850 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:41.850 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:41.850 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:41.850 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:41.850 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:41.850 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:41.850 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:41.850 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:41.850 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:41.850 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:41.850 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.847+0000 7fe468e63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:41.881 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.880+0000 7fe46159a640 1 -- 172.21.15.122:0/302781569 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe45c001b50 con 0x7fe4641430c0 2024-03-25T18:21:41.882 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.880+0000 7fe46159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:41.882 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.880+0000 7fe46159a640 1 -- 172.21.15.122:0/302781569 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fe45c001f60 con 0x7fe4641430c0 2024-03-25T18:21:41.882 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.880+0000 7fe468e63640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:41.882 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.880+0000 7fe468e63640 4 set_mon_vals no callback set 2024-03-25T18:21:41.882 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.880+0000 7fe46159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:41.882 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.880+0000 7fe46159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:41.882 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.880+0000 7fe46159a640 10 monclient: dump: 2024-03-25T18:21:41.882 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:41.882 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:41.882 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:41.882 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:41.882 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:41.883 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:41.883 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:41.883 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:41.883 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:41.883 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:41.885 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.881+0000 7fe468e63640 10 monclient: shutdown 2024-03-25T18:21:41.885 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.881+0000 7fe468e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:41.886 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.881+0000 7fe468e63640 1 -- 172.21.15.122:0/302781569 >> v1:172.21.15.122:6790/0 conn(0x7fe4641430c0 legacy=0x7fe46414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:41.886 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.881+0000 7fe468e63640 1 -- 172.21.15.122:0/302781569 shutdown_connections 2024-03-25T18:21:41.886 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.881+0000 7fe468e63640 1 -- 172.21.15.122:0/302781569 >> 172.21.15.122:0/302781569 conn(0x7fe4640b7ea0 msgr2=0x7fe4640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:41.886 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.881+0000 7fe468e63640 1 -- 172.21.15.122:0/302781569 shutdown_connections 2024-03-25T18:21:41.886 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.881+0000 7fe468e63640 1 -- 172.21.15.122:0/302781569 wait complete. 2024-03-25T18:21:41.886 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.882+0000 7fe468e63640 10 monclient: build_initial_monmap 2024-03-25T18:21:41.886 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.882+0000 7fe468e63640 10 monclient: monmap: 2024-03-25T18:21:41.886 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:21:41.886 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:41.886 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:21:41.886 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:21:41.886 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:41.887 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:21:41.887 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:21:41.887 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:21:41.887 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:21:41.887 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:41.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.883+0000 7fe468e63640 1 librados: starting msgr at 2024-03-25T18:21:41.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.883+0000 7fe468e63640 1 librados: starting objecter 2024-03-25T18:21:41.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.883+0000 7fe468e63640 1 Processor -- start 2024-03-25T18:21:41.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.883+0000 7fe468e63640 1 -- start start 2024-03-25T18:21:41.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.883+0000 7fe468e63640 1 librados: setting wanted keys 2024-03-25T18:21:41.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.883+0000 7fe468e63640 1 librados: calling monclient init 2024-03-25T18:21:41.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.883+0000 7fe468e63640 10 monclient: init 2024-03-25T18:21:41.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.883+0000 7fe468e63640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:41.888 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.883+0000 7fe468e63640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:41.888 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.883+0000 7fe468e63640 10 monclient(hunting): picked mon.noname-c con 0x7fe4640a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:21:41.888 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.883+0000 7fe468e63640 10 monclient(hunting): picked mon.noname-a con 0x7fe46413f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:21:41.888 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.883+0000 7fe468e63640 10 monclient(hunting): picked mon.noname-b con 0x7fe4641430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:21:41.888 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.884+0000 7fe468e63640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4641d73f0 con 0x7fe46413f710 2024-03-25T18:21:41.888 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.884+0000 7fe468e63640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4641395e0 con 0x7fe4641430c0 2024-03-25T18:21:41.888 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.884+0000 7fe468e63640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4641398c0 con 0x7fe4640a98f0 2024-03-25T18:21:41.888 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.884+0000 7fe468e63640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:41.888 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.884+0000 7fe468e63640 10 monclient(hunting): authenticate will time out at 7625.150054s 2024-03-25T18:21:41.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.886+0000 7fe461d9b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe46413f710 0x7fe4641d4450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:34332/0 (socket says 172.21.15.122:34332) 2024-03-25T18:21:41.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.886+0000 7fe461d9b640 1 -- 172.21.15.122:0/3194923184 learned_addr learned my addr 172.21.15.122:0/3194923184 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:21:41.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.887+0000 7fe452ffd640 10 client.?.objecter ms_handle_connect 0x7fe46413f710 2024-03-25T18:21:41.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.887+0000 7fe452ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:41.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.898+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368753134 0 0) 0x7fe4641d73f0 con 0x7fe46413f710 2024-03-25T18:21:41.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.898+0000 7fe452ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:41.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.898+0000 7fe452ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:41.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.898+0000 7fe452ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:41.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.898+0000 7fe452ffd640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:21:41.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.898+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe444004eb0 con 0x7fe46413f710 2024-03-25T18:21:41.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.898+0000 7fe452ffd640 10 client.?.objecter ms_handle_connect 0x7fe4640a98f0 2024-03-25T18:21:41.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.898+0000 7fe452ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:41.906 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.905+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1369516333 0 0) 0x7fe4641398c0 con 0x7fe4640a98f0 2024-03-25T18:21:41.906 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.905+0000 7fe452ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:41.906 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.905+0000 7fe452ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:41.906 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.905+0000 7fe452ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:41.907 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.905+0000 7fe452ffd640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:21:41.907 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.905+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4641d73f0 con 0x7fe4640a98f0 2024-03-25T18:21:41.907 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.906+0000 7fe452ffd640 10 client.?.objecter ms_handle_connect 0x7fe4641430c0 2024-03-25T18:21:41.907 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.906+0000 7fe452ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:41.911 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.910+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2527438096 0 0) 0x7fe444004eb0 con 0x7fe46413f710 2024-03-25T18:21:41.911 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.910+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe4641398c0 con 0x7fe46413f710 2024-03-25T18:21:41.920 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.919+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1720858895 0 0) 0x7fe4641d73f0 con 0x7fe4640a98f0 2024-03-25T18:21:41.920 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.919+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe444004eb0 con 0x7fe4640a98f0 2024-03-25T18:21:41.920 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.919+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fe4540018f0 con 0x7fe4640a98f0 2024-03-25T18:21:41.920 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.919+0000 7fe452ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:41.920 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.919+0000 7fe452ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:41.920 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.919+0000 7fe452ffd640 10 monclient(hunting): dump: 2024-03-25T18:21:41.921 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:41.921 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:41.921 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:41.921 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:41.921 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:41.921 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:41.921 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:41.921 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:41.921 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:41.921 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:41.921 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.919+0000 7fe452ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:41.927 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1299269181 0 0) 0x7fe444004eb0 con 0x7fe4640a98f0 2024-03-25T18:21:41.927 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 >> v1:172.21.15.204:6789/0 conn(0x7fe4641430c0 legacy=0x7fe464137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:41.927 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 >> v1:172.21.15.122:6789/0 conn(0x7fe46413f710 legacy=0x7fe4641d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:41.927 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe452ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:41.927 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe452ffd640 1 monclient: found mon.c 2024-03-25T18:21:41.927 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe452ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:41.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe452ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:21:41.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe464137b40 con 0x7fe4640a98f0 2024-03-25T18:21:41.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe452ffd640 10 monclient: _finish_auth 0 2024-03-25T18:21:41.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe452ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:41.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe452ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:41.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe468e63640 5 monclient: authenticate success, global_id 4442 2024-03-25T18:21:41.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe468e63640 10 monclient: _renew_subs 2024-03-25T18:21:41.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe468e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:21:41.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe468e63640 1 -- 172.21.15.122:0/3194923184 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4641d7f90 con 0x7fe4640a98f0 2024-03-25T18:21:41.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe468e63640 10 client.4442.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:41.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe468e63640 10 monclient: _renew_subs 2024-03-25T18:21:41.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe468e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:21:41.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe468e63640 1 -- 172.21.15.122:0/3194923184 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4641d4b60 con 0x7fe4640a98f0 2024-03-25T18:21:41.929 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.925+0000 7fe468e63640 1 librados: init done 2024-03-25T18:21:41.929 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.926+0000 7fe468e63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:41.929 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.926+0000 7fe468e63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:41.929 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.926+0000 7fe468e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:21:41.929 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:41.926+0000 7fe468e63640 1 -- 172.21.15.122:0/3194923184 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4640627b0 con 0x7fe4640a98f0 2024-03-25T18:21:42.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.223+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe454003610 con 0x7fe4640a98f0 2024-03-25T18:21:42.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.223+0000 7fe452ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:42.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.223+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fe454003a20 con 0x7fe4640a98f0 2024-03-25T18:21:42.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.223+0000 7fe452ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:42.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.223+0000 7fe452ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:42.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.223+0000 7fe452ffd640 10 monclient: dump: 2024-03-25T18:21:42.225 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:42.225 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:42.225 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:42.225 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:42.226 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:42.226 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:42.226 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:42.226 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:42.226 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:42.226 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:42.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.223+0000 7fe460d99640 4 set_mon_vals no callback set 2024-03-25T18:21:42.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.224+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fe454003c40 con 0x7fe4640a98f0 2024-03-25T18:21:42.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.225+0000 7fe461d9b640 10 monclient: get_auth_request con 0x7fe44406e020 auth_method 0 2024-03-25T18:21:42.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.225+0000 7fe452ffd640 10 client.4442.objecter ms_handle_connect 0x7fe44406e020 2024-03-25T18:21:42.233 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.232+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fe454053610 con 0x7fe4640a98f0 2024-03-25T18:21:42.233 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.232+0000 7fe452ffd640 10 client.4442.objecter ms_dispatch 0x7fe464000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:42.233 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.232+0000 7fe452ffd640 3 client.4442.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:42.233 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.232+0000 7fe452ffd640 3 client.4442.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:42.233 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.232+0000 7fe452ffd640 20 client.4442.objecter dump_active .. 0 homeless 2024-03-25T18:21:42.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.232+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe45404f120 con 0x7fe4640a98f0 2024-03-25T18:21:42.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.232+0000 7fe452ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:42.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.232+0000 7fe452ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:42.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.479+0000 7fe468e63640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:42.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.479+0000 7fe468e63640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:42.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.479+0000 7fe468e63640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:21:42.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.479+0000 7fe468e63640 1 -- 172.21.15.122:0/3194923184 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe464142290 con 0x7fe4640a98f0 2024-03-25T18:21:42.510 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.509+0000 7fe452ffd640 1 -- 172.21.15.122:0/3194923184 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fe454051050 con 0x7fe4640a98f0 2024-03-25T18:21:42.510 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.509+0000 7fe452ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:42.510 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.509+0000 7fe452ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:42.511 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:21:42.511 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:42.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.511+0000 7fe468e63640 10 librados: watch_flush enter 2024-03-25T18:21:42.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.511+0000 7fe468e63640 10 librados: watch_flush exit 2024-03-25T18:21:42.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.511+0000 7fe468e63640 20 client.4442.objecter shutdown clearing up homeless session... 2024-03-25T18:21:42.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.511+0000 7fe468e63640 10 client.4442.objecter successfully canceled tick 2024-03-25T18:21:42.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.511+0000 7fe468e63640 1 -- 172.21.15.122:0/3194923184 >> v1:172.21.15.204:6800/676228005 conn(0x7fe44406e020 legacy=0x7fe4440704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:42.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.511+0000 7fe468e63640 10 monclient: shutdown 2024-03-25T18:21:42.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.511+0000 7fe468e63640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:42.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.511+0000 7fe468e63640 1 -- 172.21.15.122:0/3194923184 >> v1:172.21.15.122:6790/0 conn(0x7fe4640a98f0 legacy=0x7fe4641d7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:42.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.511+0000 7fe468e63640 1 -- 172.21.15.122:0/3194923184 shutdown_connections 2024-03-25T18:21:42.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.511+0000 7fe468e63640 1 -- 172.21.15.122:0/3194923184 >> 172.21.15.122:0/3194923184 conn(0x7fe4640b7ea0 msgr2=0x7fe464074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:42.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.511+0000 7fe468e63640 1 -- 172.21.15.122:0/3194923184 shutdown_connections 2024-03-25T18:21:42.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.511+0000 7fe468e63640 1 -- 172.21.15.122:0/3194923184 wait complete. 2024-03-25T18:21:42.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:42.511+0000 7fe468e63640 1 librados: shutdown 2024-03-25T18:21:42.526 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:46.379 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:05:40.122 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:46.423 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:05:40.169 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:46.430 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:05:40.173 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:46.438 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:05:40.183 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:46.466 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:05:40.209 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:46.467 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:05:40.213 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:46.495 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:05:40.238 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:46.499 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:05:40.242 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:46.529 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:05:40.271 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:49.333 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:49.447 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.446+0000 7fb54b17b640 10 monclient: get_monmap_and_config 2024-03-25T18:21:49.448 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.446+0000 7fb54b17b640 10 monclient: build_initial_monmap 2024-03-25T18:21:49.448 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.446+0000 7fb54b17b640 10 monclient: monmap: 2024-03-25T18:21:49.448 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:21:49.448 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:49.448 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:21:49.447877+0000 2024-03-25T18:21:49.448 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:21:49.447877+0000 2024-03-25T18:21:49.448 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:49.448 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:49.448 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:21:49.448 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:21:49.448 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:21:49.448 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:49.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.447+0000 7fb54b17b640 1 Processor -- start 2024-03-25T18:21:49.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.447+0000 7fb54b17b640 1 -- start start 2024-03-25T18:21:49.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.447+0000 7fb54b17b640 10 monclient: init 2024-03-25T18:21:49.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.448+0000 7fb54b17b640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:49.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.448+0000 7fb54b17b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:49.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.448+0000 7fb54b17b640 10 monclient(hunting): picked mon.noname-b con 0x7fb5440b4d60 addr v1:172.21.15.122:6790/0 2024-03-25T18:21:49.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.448+0000 7fb54b17b640 10 monclient(hunting): picked mon.noname-c con 0x7fb5440b1730 addr v1:172.21.15.204:6789/0 2024-03-25T18:21:49.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.448+0000 7fb54b17b640 10 monclient(hunting): picked mon.noname-a con 0x7fb54414be90 addr v1:172.21.15.122:6789/0 2024-03-25T18:21:49.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.448+0000 7fb54b17b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5440af940 con 0x7fb54414be90 2024-03-25T18:21:49.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.448+0000 7fb54b17b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5440b0a70 con 0x7fb5440b1730 2024-03-25T18:21:49.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.448+0000 7fb54b17b640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5440b0d50 con 0x7fb5440b4d60 2024-03-25T18:21:49.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.448+0000 7fb54b17b640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:49.451 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.448+0000 7fb54b17b640 10 monclient(hunting): authenticate will time out at 7632.714579s 2024-03-25T18:21:49.462 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.461+0000 7fb543fff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fb5440b1730 0x7fb5440b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:41358/0 (socket says 172.21.15.122:41358) 2024-03-25T18:21:49.462 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.461+0000 7fb543fff640 1 -- 172.21.15.122:0/2420075771 learned_addr learned my addr 172.21.15.122:0/2420075771 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:21:49.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.481+0000 7fb5437fe640 1 -- 172.21.15.122:0/2420075771 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1534909171 0 0) 0x7fb5440af940 con 0x7fb54414be90 2024-03-25T18:21:49.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.481+0000 7fb5437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:49.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.481+0000 7fb5437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:49.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.481+0000 7fb5437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:49.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.481+0000 7fb5437fe640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:21:49.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.481+0000 7fb5437fe640 1 -- 172.21.15.122:0/2420075771 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb534003520 con 0x7fb54414be90 2024-03-25T18:21:49.484 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.482+0000 7fb5437fe640 1 -- 172.21.15.122:0/2420075771 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 294473127 0 0) 0x7fb5440b0a70 con 0x7fb5440b1730 2024-03-25T18:21:49.484 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.482+0000 7fb5437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:49.484 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.482+0000 7fb5437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:49.484 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.482+0000 7fb5437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:49.484 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.482+0000 7fb5437fe640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:21:49.484 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.482+0000 7fb5437fe640 1 -- 172.21.15.122:0/2420075771 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5440af940 con 0x7fb5440b1730 2024-03-25T18:21:49.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb5437fe640 1 -- 172.21.15.122:0/2420075771 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 637568763 0 0) 0x7fb5440af940 con 0x7fb5440b1730 2024-03-25T18:21:49.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb5437fe640 1 -- 172.21.15.122:0/2420075771 >> v1:172.21.15.122:6790/0 conn(0x7fb5440b4d60 legacy=0x7fb54413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:49.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb5437fe640 1 -- 172.21.15.122:0/2420075771 >> v1:172.21.15.122:6789/0 conn(0x7fb54414be90 legacy=0x7fb54414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:49.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb5437fe640 10 monclient: _finish_hunting 0 2024-03-25T18:21:49.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb5437fe640 1 monclient: found mon.noname-c 2024-03-25T18:21:49.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb5437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:49.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb5437fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:21:49.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb5437fe640 1 -- 172.21.15.122:0/2420075771 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5440b1030 con 0x7fb5440b1730 2024-03-25T18:21:49.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb5437fe640 10 monclient: _finish_auth 0 2024-03-25T18:21:49.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb5437fe640 10 monclient: _check_auth_tickets 2024-03-25T18:21:49.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb5437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:49.495 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb5437fe640 1 -- 172.21.15.122:0/2420075771 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fb530001670 con 0x7fb5440b1730 2024-03-25T18:21:49.495 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb54b17b640 5 monclient: authenticate success, global_id 4468 2024-03-25T18:21:49.495 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb54b17b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:49.495 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb5437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:49.495 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb5437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:49.495 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb5437fe640 10 monclient: dump: 2024-03-25T18:21:49.495 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:49.495 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:49.495 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:49.495 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:49.495 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:49.495 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:49.495 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:49.495 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:49.495 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:49.496 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:49.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.492+0000 7fb54b17b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:49.502 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.500+0000 7fb5437fe640 1 -- 172.21.15.122:0/2420075771 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb5300017d0 con 0x7fb5440b1730 2024-03-25T18:21:49.502 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.500+0000 7fb5437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:49.502 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.500+0000 7fb5437fe640 1 -- 172.21.15.122:0/2420075771 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fb530001ba0 con 0x7fb5440b1730 2024-03-25T18:21:49.502 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.501+0000 7fb54b17b640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:49.502 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.501+0000 7fb54b17b640 4 set_mon_vals no callback set 2024-03-25T18:21:49.502 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.501+0000 7fb5437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:49.502 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.501+0000 7fb5437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:49.502 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.501+0000 7fb5437fe640 10 monclient: dump: 2024-03-25T18:21:49.503 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:49.503 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:49.503 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:49.503 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:49.503 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:49.503 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:49.503 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:49.503 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:49.503 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:49.503 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:49.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.501+0000 7fb54b17b640 10 monclient: shutdown 2024-03-25T18:21:49.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.501+0000 7fb54b17b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:49.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.501+0000 7fb54b17b640 1 -- 172.21.15.122:0/2420075771 >> v1:172.21.15.204:6789/0 conn(0x7fb5440b1730 legacy=0x7fb5440b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:49.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.501+0000 7fb54b17b640 1 -- 172.21.15.122:0/2420075771 shutdown_connections 2024-03-25T18:21:49.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.501+0000 7fb54b17b640 1 -- 172.21.15.122:0/2420075771 >> 172.21.15.122:0/2420075771 conn(0x7fb5440617a0 msgr2=0x7fb544061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:49.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.502+0000 7fb54b17b640 1 -- 172.21.15.122:0/2420075771 shutdown_connections 2024-03-25T18:21:49.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.502+0000 7fb54b17b640 1 -- 172.21.15.122:0/2420075771 wait complete. 2024-03-25T18:21:49.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.502+0000 7fb54b17b640 10 monclient: build_initial_monmap 2024-03-25T18:21:49.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.502+0000 7fb54b17b640 10 monclient: monmap: 2024-03-25T18:21:49.506 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:21:49.506 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:49.506 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:21:49.507 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:21:49.507 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:49.507 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:21:49.507 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:21:49.507 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:21:49.507 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:21:49.507 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:49.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.503+0000 7fb54b17b640 1 librados: starting msgr at 2024-03-25T18:21:49.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.503+0000 7fb54b17b640 1 librados: starting objecter 2024-03-25T18:21:49.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.503+0000 7fb54b17b640 1 Processor -- start 2024-03-25T18:21:49.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.503+0000 7fb54b17b640 1 -- start start 2024-03-25T18:21:49.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.503+0000 7fb54b17b640 1 librados: setting wanted keys 2024-03-25T18:21:49.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.503+0000 7fb54b17b640 1 librados: calling monclient init 2024-03-25T18:21:49.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.503+0000 7fb54b17b640 10 monclient: init 2024-03-25T18:21:49.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.504+0000 7fb54b17b640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:49.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.504+0000 7fb54b17b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:49.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.504+0000 7fb54b17b640 10 monclient(hunting): picked mon.noname-b con 0x7fb5440b1730 addr v1:172.21.15.204:6789/0 2024-03-25T18:21:49.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.504+0000 7fb54b17b640 10 monclient(hunting): picked mon.noname-a con 0x7fb5440b4d60 addr v1:172.21.15.122:6789/0 2024-03-25T18:21:49.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.504+0000 7fb54b17b640 10 monclient(hunting): picked mon.noname-c con 0x7fb54414be90 addr v1:172.21.15.122:6790/0 2024-03-25T18:21:49.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.504+0000 7fb54b17b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb544142610 con 0x7fb5440b4d60 2024-03-25T18:21:49.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.504+0000 7fb54b17b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb544136330 con 0x7fb5440b1730 2024-03-25T18:21:49.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.504+0000 7fb54b17b640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb544136610 con 0x7fb54414be90 2024-03-25T18:21:49.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.504+0000 7fb54b17b640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:49.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.504+0000 7fb54b17b640 10 monclient(hunting): authenticate will time out at 7632.770562s 2024-03-25T18:21:49.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.512+0000 7fb549718640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7fb54414be90 0x7fb544140380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:53522/0 (socket says 172.21.15.122:53522) 2024-03-25T18:21:49.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.512+0000 7fb549718640 1 -- 172.21.15.122:0/797124267 learned_addr learned my addr 172.21.15.122:0/797124267 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:21:49.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.513+0000 7fb5417fa640 10 client.?.objecter ms_handle_connect 0x7fb54414be90 2024-03-25T18:21:49.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.513+0000 7fb5417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:49.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.518+0000 7fb5417fa640 10 client.?.objecter ms_handle_connect 0x7fb5440b4d60 2024-03-25T18:21:49.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.518+0000 7fb5417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:49.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.519+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025114321 0 0) 0x7fb544136610 con 0x7fb54414be90 2024-03-25T18:21:49.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.519+0000 7fb5417fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:49.521 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.519+0000 7fb5417fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:49.521 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.519+0000 7fb5417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:49.521 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.519+0000 7fb5417fa640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:21:49.521 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.519+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb528004eb0 con 0x7fb54414be90 2024-03-25T18:21:49.524 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.523+0000 7fb5417fa640 10 client.?.objecter ms_handle_connect 0x7fb5440b1730 2024-03-25T18:21:49.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.523+0000 7fb5417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:49.527 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.525+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292181301 0 0) 0x7fb544142610 con 0x7fb5440b4d60 2024-03-25T18:21:49.527 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.526+0000 7fb5417fa640 10 monclient(hunting): _negotiate 2024-03-25T18:21:49.527 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.526+0000 7fb5417fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:49.527 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.526+0000 7fb5417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:49.527 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.526+0000 7fb5417fa640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:21:49.527 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.526+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb544136610 con 0x7fb5440b4d60 2024-03-25T18:21:49.542 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.540+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2009897003 0 0) 0x7fb528004eb0 con 0x7fb54414be90 2024-03-25T18:21:49.542 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.540+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb544142610 con 0x7fb54414be90 2024-03-25T18:21:49.542 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.540+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fb53c001c70 con 0x7fb54414be90 2024-03-25T18:21:49.542 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.540+0000 7fb5417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:49.542 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.540+0000 7fb5417fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:49.542 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.540+0000 7fb5417fa640 10 monclient(hunting): dump: 2024-03-25T18:21:49.542 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:49.542 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:49.542 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:49.542 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:49.543 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:49.543 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:49.543 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:49.543 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:49.543 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:49.543 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:49.543 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.540+0000 7fb5417fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:49.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.552+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1312255054 0 0) 0x7fb544136610 con 0x7fb5440b4d60 2024-03-25T18:21:49.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.552+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb528004eb0 con 0x7fb5440b4d60 2024-03-25T18:21:49.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.552+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fb530001ae0 con 0x7fb5440b4d60 2024-03-25T18:21:49.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.552+0000 7fb5417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:49.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.552+0000 7fb5417fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:49.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.552+0000 7fb5417fa640 10 monclient(hunting): dump: 2024-03-25T18:21:49.554 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:49.554 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:49.554 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:49.554 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:49.554 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:49.554 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:49.554 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:49.555 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:49.555 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:49.555 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:49.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2764860810 0 0) 0x7fb544142610 con 0x7fb54414be90 2024-03-25T18:21:49.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 >> v1:172.21.15.204:6789/0 conn(0x7fb5440b1730 legacy=0x7fb544138ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:49.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 >> v1:172.21.15.122:6789/0 conn(0x7fb5440b4d60 legacy=0x7fb544134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:49.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb5417fa640 10 monclient: _finish_hunting 0 2024-03-25T18:21:49.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb5417fa640 1 monclient: found mon.c 2024-03-25T18:21:49.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb5417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:49.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb5417fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:21:49.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5441355c0 con 0x7fb54414be90 2024-03-25T18:21:49.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb5417fa640 10 monclient: _finish_auth 0 2024-03-25T18:21:49.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb5417fa640 10 monclient: _check_auth_tickets 2024-03-25T18:21:49.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb5417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:49.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb54b17b640 5 monclient: authenticate success, global_id 4448 2024-03-25T18:21:49.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb54b17b640 10 monclient: _renew_subs 2024-03-25T18:21:49.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb54b17b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:21:49.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb54b17b640 1 -- 172.21.15.122:0/797124267 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb544135810 con 0x7fb54414be90 2024-03-25T18:21:49.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb54b17b640 10 client.4448.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:49.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb54b17b640 10 monclient: _renew_subs 2024-03-25T18:21:49.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb54b17b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:21:49.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb54b17b640 1 -- 172.21.15.122:0/797124267 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb544143240 con 0x7fb54414be90 2024-03-25T18:21:49.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.554+0000 7fb54b17b640 1 librados: init done 2024-03-25T18:21:49.558 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.555+0000 7fb54b17b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:49.558 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.555+0000 7fb54b17b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:49.558 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.555+0000 7fb54b17b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:21:49.558 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.555+0000 7fb54b17b640 1 -- 172.21.15.122:0/797124267 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5441401f0 con 0x7fb54414be90 2024-03-25T18:21:49.580 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.579+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb53c003990 con 0x7fb54414be90 2024-03-25T18:21:49.580 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.579+0000 7fb5417fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:49.580 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.579+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fb53c003d20 con 0x7fb54414be90 2024-03-25T18:21:49.580 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.579+0000 7fb5417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:49.580 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.579+0000 7fb5417fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:49.580 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.579+0000 7fb5417fa640 10 monclient: dump: 2024-03-25T18:21:49.580 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:49.580 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:49.581 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:49.581 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:49.581 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:49.581 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:49.581 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:49.581 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:49.581 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:49.581 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:49.581 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.579+0000 7fb542ffd640 4 set_mon_vals no callback set 2024-03-25T18:21:49.582 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.579+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fb53c003f40 con 0x7fb54414be90 2024-03-25T18:21:49.582 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.580+0000 7fb548f17640 10 monclient: get_auth_request con 0x7fb52806de90 auth_method 0 2024-03-25T18:21:49.582 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.580+0000 7fb5417fa640 10 client.4448.objecter ms_handle_connect 0x7fb52806de90 2024-03-25T18:21:49.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.583+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fb53c0549a0 con 0x7fb54414be90 2024-03-25T18:21:49.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.583+0000 7fb5417fa640 10 client.4448.objecter ms_dispatch 0x7fb544000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:49.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.583+0000 7fb5417fa640 3 client.4448.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:49.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.583+0000 7fb5417fa640 3 client.4448.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:49.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.583+0000 7fb5417fa640 20 client.4448.objecter dump_active .. 0 homeless 2024-03-25T18:21:49.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.893+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb53c0504b0 con 0x7fb54414be90 2024-03-25T18:21:49.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.893+0000 7fb5417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:49.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:49.893+0000 7fb5417fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:50.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.148+0000 7fb54b17b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:50.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.148+0000 7fb54b17b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:50.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.148+0000 7fb54b17b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:21:50.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.148+0000 7fb54b17b640 1 -- 172.21.15.122:0/797124267 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb544136850 con 0x7fb54414be90 2024-03-25T18:21:50.205 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.203+0000 7fb5417fa640 1 -- 172.21.15.122:0/797124267 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fb544136850 con 0x7fb54414be90 2024-03-25T18:21:50.205 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.203+0000 7fb5417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:50.205 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.203+0000 7fb5417fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:50.205 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:21:50.205 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:50.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.205+0000 7fb54b17b640 10 librados: watch_flush enter 2024-03-25T18:21:50.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.206+0000 7fb54b17b640 10 librados: watch_flush exit 2024-03-25T18:21:50.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.206+0000 7fb54b17b640 20 client.4448.objecter shutdown clearing up homeless session... 2024-03-25T18:21:50.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.206+0000 7fb54b17b640 10 client.4448.objecter successfully canceled tick 2024-03-25T18:21:50.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.206+0000 7fb54b17b640 1 -- 172.21.15.122:0/797124267 >> v1:172.21.15.204:6800/676228005 conn(0x7fb52806de90 legacy=0x7fb528070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:50.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.206+0000 7fb54b17b640 10 monclient: shutdown 2024-03-25T18:21:50.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.206+0000 7fb54b17b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:50.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.206+0000 7fb54b17b640 1 -- 172.21.15.122:0/797124267 >> v1:172.21.15.122:6790/0 conn(0x7fb54414be90 legacy=0x7fb544140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:50.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.206+0000 7fb54b17b640 1 -- 172.21.15.122:0/797124267 shutdown_connections 2024-03-25T18:21:50.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.206+0000 7fb54b17b640 1 -- 172.21.15.122:0/797124267 >> 172.21.15.122:0/797124267 conn(0x7fb5440617a0 msgr2=0x7fb544061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:50.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.206+0000 7fb54b17b640 1 -- 172.21.15.122:0/797124267 shutdown_connections 2024-03-25T18:21:50.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.206+0000 7fb54b17b640 1 -- 172.21.15.122:0/797124267 wait complete. 2024-03-25T18:21:50.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:50.206+0000 7fb54b17b640 1 librados: shutdown 2024-03-25T18:21:50.219 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:57.026 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:57.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.137+0000 7f2e94284640 10 monclient: get_monmap_and_config 2024-03-25T18:21:57.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.137+0000 7f2e94284640 10 monclient: build_initial_monmap 2024-03-25T18:21:57.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.137+0000 7f2e94284640 10 monclient: monmap: 2024-03-25T18:21:57.139 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:21:57.139 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:57.139 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:21:57.138921+0000 2024-03-25T18:21:57.139 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:21:57.138921+0000 2024-03-25T18:21:57.139 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:57.139 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:57.139 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:21:57.139 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:21:57.140 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:21:57.140 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:57.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.138+0000 7f2e94284640 1 Processor -- start 2024-03-25T18:21:57.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.138+0000 7f2e94284640 1 -- start start 2024-03-25T18:21:57.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.138+0000 7f2e94284640 10 monclient: init 2024-03-25T18:21:57.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.139+0000 7f2e94284640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:57.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.139+0000 7f2e94284640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:57.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.139+0000 7f2e94284640 10 monclient(hunting): picked mon.noname-a con 0x7f2e8c0a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:21:57.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.139+0000 7f2e94284640 10 monclient(hunting): picked mon.noname-b con 0x7f2e8c13f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:21:57.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.139+0000 7f2e94284640 10 monclient(hunting): picked mon.noname-c con 0x7f2e8c1430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:21:57.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.139+0000 7f2e94284640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8c142520 con 0x7f2e8c0a98f0 2024-03-25T18:21:57.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.139+0000 7f2e94284640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8c0b28a0 con 0x7f2e8c1430c0 2024-03-25T18:21:57.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.139+0000 7f2e94284640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8c0b2a30 con 0x7f2e8c13f710 2024-03-25T18:21:57.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.139+0000 7f2e94284640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:57.141 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.139+0000 7f2e94284640 10 monclient(hunting): authenticate will time out at 7640.405874s 2024-03-25T18:21:57.147 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.146+0000 7f2e9181f640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f2e8c13f710 0x7f2e8c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:35576/0 (socket says 172.21.15.122:35576) 2024-03-25T18:21:57.147 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.146+0000 7f2e9181f640 1 -- 172.21.15.122:0/95645893 learned_addr learned my addr 172.21.15.122:0/95645893 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:21:57.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.162+0000 7f2e9101e640 1 -- 172.21.15.122:0/95645893 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 790703617 0 0) 0x7f2e8c0b2a30 con 0x7f2e8c13f710 2024-03-25T18:21:57.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.162+0000 7f2e9101e640 10 monclient(hunting): _negotiate 2024-03-25T18:21:57.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.162+0000 7f2e9101e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:57.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.162+0000 7f2e9101e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:57.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.162+0000 7f2e9101e640 10 monclient(hunting): my global_id is 4451 2024-03-25T18:21:57.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.162+0000 7f2e9101e640 1 -- 172.21.15.122:0/95645893 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e78003520 con 0x7f2e8c13f710 2024-03-25T18:21:57.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.166+0000 7f2e9101e640 1 -- 172.21.15.122:0/95645893 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1074489580 0 0) 0x7f2e8c142520 con 0x7f2e8c0a98f0 2024-03-25T18:21:57.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.166+0000 7f2e9101e640 10 monclient(hunting): _negotiate 2024-03-25T18:21:57.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.166+0000 7f2e9101e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:57.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.166+0000 7f2e9101e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:57.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.166+0000 7f2e9101e640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:21:57.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.166+0000 7f2e9101e640 1 -- 172.21.15.122:0/95645893 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e8c0b2a30 con 0x7f2e8c0a98f0 2024-03-25T18:21:57.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.171+0000 7f2e9101e640 1 -- 172.21.15.122:0/95645893 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221143156 0 0) 0x7f2e8c0b28a0 con 0x7f2e8c1430c0 2024-03-25T18:21:57.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.171+0000 7f2e9101e640 10 monclient(hunting): _negotiate 2024-03-25T18:21:57.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.171+0000 7f2e9101e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:57.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.171+0000 7f2e9101e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:57.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.171+0000 7f2e9101e640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:21:57.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.171+0000 7f2e9101e640 1 -- 172.21.15.122:0/95645893 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e780039d0 con 0x7f2e8c1430c0 2024-03-25T18:21:57.175 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.173+0000 7f2e9101e640 1 -- 172.21.15.122:0/95645893 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3671474785 0 0) 0x7f2e78003520 con 0x7f2e8c13f710 2024-03-25T18:21:57.175 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.173+0000 7f2e9101e640 1 -- 172.21.15.122:0/95645893 >> v1:172.21.15.204:6789/0 conn(0x7f2e8c1430c0 legacy=0x7f2e8c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:57.175 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.173+0000 7f2e9101e640 1 -- 172.21.15.122:0/95645893 >> v1:172.21.15.122:6789/0 conn(0x7f2e8c0a98f0 legacy=0x7f2e8c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:57.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.173+0000 7f2e9101e640 10 monclient: _finish_hunting 0 2024-03-25T18:21:57.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.173+0000 7f2e9101e640 1 monclient: found mon.noname-b 2024-03-25T18:21:57.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.173+0000 7f2e9101e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:57.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.173+0000 7f2e9101e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:21:57.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.173+0000 7f2e9101e640 1 -- 172.21.15.122:0/95645893 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8c0b44c0 con 0x7f2e8c13f710 2024-03-25T18:21:57.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.173+0000 7f2e9101e640 10 monclient: _finish_auth 0 2024-03-25T18:21:57.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.173+0000 7f2e9101e640 10 monclient: _check_auth_tickets 2024-03-25T18:21:57.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.173+0000 7f2e9101e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:57.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.174+0000 7f2e9101e640 1 -- 172.21.15.122:0/95645893 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f2e7c001690 con 0x7f2e8c13f710 2024-03-25T18:21:57.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.174+0000 7f2e9101e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:57.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.174+0000 7f2e9101e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:57.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.174+0000 7f2e9101e640 10 monclient: dump: 2024-03-25T18:21:57.176 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:57.176 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:57.176 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:57.177 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:57.177 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:57.177 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:57.177 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:57.177 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:57.177 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:57.177 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:57.177 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.174+0000 7f2e94284640 5 monclient: authenticate success, global_id 4451 2024-03-25T18:21:57.177 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.174+0000 7f2e94284640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:57.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.180+0000 7f2e9101e640 1 -- 172.21.15.122:0/95645893 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e7c001870 con 0x7f2e8c13f710 2024-03-25T18:21:57.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.180+0000 7f2e9101e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:57.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.180+0000 7f2e9101e640 1 -- 172.21.15.122:0/95645893 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f2e7c001c80 con 0x7f2e8c13f710 2024-03-25T18:21:57.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.180+0000 7f2e94284640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:57.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.180+0000 7f2e94284640 4 set_mon_vals no callback set 2024-03-25T18:21:57.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.180+0000 7f2e9101e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:57.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.180+0000 7f2e9101e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:57.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.180+0000 7f2e9101e640 10 monclient: dump: 2024-03-25T18:21:57.182 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:57.182 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:57.182 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:57.182 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:57.183 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:57.183 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:57.183 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:57.183 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:57.183 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:57.183 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:57.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.181+0000 7f2e94284640 10 monclient: shutdown 2024-03-25T18:21:57.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.181+0000 7f2e94284640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:57.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.181+0000 7f2e94284640 1 -- 172.21.15.122:0/95645893 >> v1:172.21.15.122:6790/0 conn(0x7f2e8c13f710 legacy=0x7f2e8c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:57.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.181+0000 7f2e94284640 1 -- 172.21.15.122:0/95645893 shutdown_connections 2024-03-25T18:21:57.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.181+0000 7f2e94284640 1 -- 172.21.15.122:0/95645893 >> 172.21.15.122:0/95645893 conn(0x7f2e8c0b7ea0 msgr2=0x7f2e8c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:57.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.182+0000 7f2e94284640 1 -- 172.21.15.122:0/95645893 shutdown_connections 2024-03-25T18:21:57.185 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.182+0000 7f2e94284640 1 -- 172.21.15.122:0/95645893 wait complete. 2024-03-25T18:21:57.185 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.183+0000 7f2e94284640 10 monclient: build_initial_monmap 2024-03-25T18:21:57.185 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.183+0000 7f2e94284640 10 monclient: monmap: 2024-03-25T18:21:57.185 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:21:57.185 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:57.185 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:21:57.185 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:21:57.185 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:57.185 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:21:57.185 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:21:57.185 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:21:57.185 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:21:57.185 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:57.185 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.183+0000 7f2e94284640 1 librados: starting msgr at 2024-03-25T18:21:57.185 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.183+0000 7f2e94284640 1 librados: starting objecter 2024-03-25T18:21:57.186 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.183+0000 7f2e94284640 1 Processor -- start 2024-03-25T18:21:57.186 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.183+0000 7f2e94284640 1 -- start start 2024-03-25T18:21:57.186 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.183+0000 7f2e94284640 1 librados: setting wanted keys 2024-03-25T18:21:57.186 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.183+0000 7f2e94284640 1 librados: calling monclient init 2024-03-25T18:21:57.186 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.183+0000 7f2e94284640 10 monclient: init 2024-03-25T18:21:57.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.184+0000 7f2e94284640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:57.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.184+0000 7f2e94284640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:57.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.184+0000 7f2e94284640 10 monclient(hunting): picked mon.noname-a con 0x7f2e8c0a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:21:57.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.185+0000 7f2e94284640 10 monclient(hunting): picked mon.noname-c con 0x7f2e8c13f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:21:57.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.185+0000 7f2e94284640 10 monclient(hunting): picked mon.noname-b con 0x7f2e8c1430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:21:57.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.185+0000 7f2e94284640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8c1d85f0 con 0x7f2e8c0a98f0 2024-03-25T18:21:57.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.185+0000 7f2e94284640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8c137b00 con 0x7f2e8c1430c0 2024-03-25T18:21:57.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.185+0000 7f2e94284640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e8c1d76e0 con 0x7f2e8c13f710 2024-03-25T18:21:57.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.185+0000 7f2e94284640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:57.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.185+0000 7f2e94284640 10 monclient(hunting): authenticate will time out at 7640.451190s 2024-03-25T18:21:57.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.199+0000 7f2e92020640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f2e8c0a98f0 0x7f2e8c1d7ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50952/0 (socket says 172.21.15.122:50952) 2024-03-25T18:21:57.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.199+0000 7f2e92020640 1 -- 172.21.15.122:0/552882237 learned_addr learned my addr 172.21.15.122:0/552882237 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:21:57.201 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.200+0000 7f2e82ffd640 10 client.?.objecter ms_handle_connect 0x7f2e8c0a98f0 2024-03-25T18:21:57.201 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.200+0000 7f2e82ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:57.204 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.203+0000 7f2e82ffd640 10 client.?.objecter ms_handle_connect 0x7f2e8c13f710 2024-03-25T18:21:57.204 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.203+0000 7f2e82ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:57.210 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.209+0000 7f2e82ffd640 10 client.?.objecter ms_handle_connect 0x7f2e8c1430c0 2024-03-25T18:21:57.211 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.209+0000 7f2e82ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:57.212 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.211+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920697229 0 0) 0x7f2e8c1d76e0 con 0x7f2e8c13f710 2024-03-25T18:21:57.212 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.211+0000 7f2e82ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:57.212 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.211+0000 7f2e82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:57.212 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.211+0000 7f2e82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:57.212 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.211+0000 7f2e82ffd640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:21:57.213 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.211+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e70004eb0 con 0x7f2e8c13f710 2024-03-25T18:21:57.214 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.213+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3126271630 0 0) 0x7f2e8c137b00 con 0x7f2e8c1430c0 2024-03-25T18:21:57.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.213+0000 7f2e82ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:57.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.213+0000 7f2e82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:57.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.213+0000 7f2e82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:57.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.213+0000 7f2e82ffd640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:21:57.215 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.213+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e8c1d76e0 con 0x7f2e8c1430c0 2024-03-25T18:21:57.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.214+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366847329 0 0) 0x7f2e8c1d85f0 con 0x7f2e8c0a98f0 2024-03-25T18:21:57.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.214+0000 7f2e82ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:57.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.214+0000 7f2e82ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:57.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.214+0000 7f2e82ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:57.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.214+0000 7f2e82ffd640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:21:57.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.214+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e8c137b00 con 0x7f2e8c0a98f0 2024-03-25T18:21:57.223 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.221+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 598855619 0 0) 0x7f2e8c1d76e0 con 0x7f2e8c1430c0 2024-03-25T18:21:57.223 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.221+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e8c1d85f0 con 0x7f2e8c1430c0 2024-03-25T18:21:57.223 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.221+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f2e84001b90 con 0x7f2e8c1430c0 2024-03-25T18:21:57.223 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.221+0000 7f2e82ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:57.223 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.221+0000 7f2e82ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:57.223 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.221+0000 7f2e82ffd640 10 monclient(hunting): dump: 2024-03-25T18:21:57.223 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:57.223 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:57.223 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:57.223 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:57.223 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:57.223 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:57.223 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:57.224 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:57.224 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:57.224 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:57.224 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.221+0000 7f2e82ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:57.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.225+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 238863379 0 0) 0x7f2e70004eb0 con 0x7f2e8c13f710 2024-03-25T18:21:57.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.225+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e8c1d76e0 con 0x7f2e8c13f710 2024-03-25T18:21:57.230 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4043234908 0 0) 0x7f2e8c1d85f0 con 0x7f2e8c1430c0 2024-03-25T18:21:57.230 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 >> v1:172.21.15.122:6790/0 conn(0x7f2e8c13f710 legacy=0x7f2e8c1384e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:57.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 >> v1:172.21.15.122:6789/0 conn(0x7f2e8c0a98f0 legacy=0x7f2e8c1d7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:57.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e82ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:57.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e82ffd640 1 monclient: found mon.b 2024-03-25T18:21:57.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e82ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:57.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e82ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:57.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e8c1d79c0 con 0x7f2e8c1430c0 2024-03-25T18:21:57.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e82ffd640 10 monclient: _finish_auth 0 2024-03-25T18:21:57.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e82ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:57.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e82ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:57.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e94284640 5 monclient: authenticate success, global_id 4477 2024-03-25T18:21:57.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e94284640 10 monclient: _renew_subs 2024-03-25T18:21:57.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e94284640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:57.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e94284640 1 -- 172.21.15.122:0/552882237 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e8c1d51f0 con 0x7f2e8c1430c0 2024-03-25T18:21:57.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e94284640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:57.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e94284640 10 monclient: _renew_subs 2024-03-25T18:21:57.232 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e94284640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:57.232 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e94284640 1 -- 172.21.15.122:0/552882237 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e8c1e3100 con 0x7f2e8c1430c0 2024-03-25T18:21:57.232 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.229+0000 7f2e94284640 1 librados: init done 2024-03-25T18:21:57.232 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.230+0000 7f2e94284640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:57.232 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.230+0000 7f2e94284640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:57.232 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.230+0000 7f2e94284640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:57.232 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.230+0000 7f2e94284640 1 -- 172.21.15.122:0/552882237 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e8c0627b0 con 0x7f2e8c1430c0 2024-03-25T18:21:57.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.513+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e84002040 con 0x7f2e8c1430c0 2024-03-25T18:21:57.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.513+0000 7f2e82ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:57.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.513+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f2e84003b30 con 0x7f2e8c1430c0 2024-03-25T18:21:57.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.513+0000 7f2e82ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:57.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.513+0000 7f2e9081d640 4 set_mon_vals no callback set 2024-03-25T18:21:57.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.513+0000 7f2e82ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:57.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.513+0000 7f2e82ffd640 10 monclient: dump: 2024-03-25T18:21:57.515 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:21:57.515 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:21:57.515 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:57.515 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:21:57.515 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:57.515 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:21:57.515 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:21:57.515 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:21:57.515 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:21:57.516 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:21:57.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.515+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f2e84019b40 con 0x7f2e8c1430c0 2024-03-25T18:21:57.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.515+0000 7f2e92020640 10 monclient: get_auth_request con 0x7f2e7006e070 auth_method 0 2024-03-25T18:21:57.517 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.516+0000 7f2e82ffd640 10 client.4477.objecter ms_handle_connect 0x7f2e7006e070 2024-03-25T18:21:57.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.520+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f2e840537e0 con 0x7f2e8c1430c0 2024-03-25T18:21:57.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.520+0000 7f2e82ffd640 10 client.4477.objecter ms_dispatch 0x7f2e8c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:57.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.520+0000 7f2e82ffd640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:57.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.520+0000 7f2e82ffd640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:57.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.520+0000 7f2e82ffd640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-25T18:21:57.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.520+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e8404f2f0 con 0x7f2e8c1430c0 2024-03-25T18:21:57.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.520+0000 7f2e82ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:57.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.520+0000 7f2e82ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:57.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.764+0000 7f2e94284640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:57.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.764+0000 7f2e94284640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:57.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.764+0000 7f2e94284640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:21:57.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.764+0000 7f2e94284640 1 -- 172.21.15.122:0/552882237 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e8c1d7b50 con 0x7f2e8c1430c0 2024-03-25T18:21:57.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.788+0000 7f2e82ffd640 1 -- 172.21.15.122:0/552882237 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f2e84051220 con 0x7f2e8c1430c0 2024-03-25T18:21:57.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.788+0000 7f2e82ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:57.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.788+0000 7f2e82ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:57.789 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:21:57.789 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:57.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.790+0000 7f2e94284640 10 librados: watch_flush enter 2024-03-25T18:21:57.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.790+0000 7f2e94284640 10 librados: watch_flush exit 2024-03-25T18:21:57.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.790+0000 7f2e94284640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-25T18:21:57.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.790+0000 7f2e94284640 10 client.4477.objecter successfully canceled tick 2024-03-25T18:21:57.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.790+0000 7f2e94284640 1 -- 172.21.15.122:0/552882237 >> v1:172.21.15.204:6800/676228005 conn(0x7f2e7006e070 legacy=0x7f2e70070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:57.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.790+0000 7f2e94284640 10 monclient: shutdown 2024-03-25T18:21:57.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.790+0000 7f2e94284640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:57.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.790+0000 7f2e94284640 1 -- 172.21.15.122:0/552882237 >> v1:172.21.15.204:6789/0 conn(0x7f2e8c1430c0 legacy=0x7f2e8c138bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:57.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.790+0000 7f2e94284640 1 -- 172.21.15.122:0/552882237 shutdown_connections 2024-03-25T18:21:57.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.790+0000 7f2e94284640 1 -- 172.21.15.122:0/552882237 >> 172.21.15.122:0/552882237 conn(0x7f2e8c0b7ea0 msgr2=0x7f2e8c14e930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:57.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.790+0000 7f2e94284640 1 -- 172.21.15.122:0/552882237 shutdown_connections 2024-03-25T18:21:57.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.790+0000 7f2e94284640 1 -- 172.21.15.122:0/552882237 wait complete. 2024-03-25T18:21:57.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:21:57.790+0000 7f2e94284640 1 librados: shutdown 2024-03-25T18:21:57.803 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:04.609 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:04.722 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.721+0000 7fe113cd0640 10 monclient: get_monmap_and_config 2024-03-25T18:22:04.722 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.721+0000 7fe113cd0640 10 monclient: build_initial_monmap 2024-03-25T18:22:04.722 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.721+0000 7fe113cd0640 10 monclient: monmap: 2024-03-25T18:22:04.722 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:22:04.722 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:04.722 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:22:04.722285+0000 2024-03-25T18:22:04.722 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:22:04.722285+0000 2024-03-25T18:22:04.722 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:04.723 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:04.723 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:22:04.723 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:22:04.723 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:22:04.723 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:04.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.721+0000 7fe113cd0640 1 Processor -- start 2024-03-25T18:22:04.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.721+0000 7fe113cd0640 1 -- start start 2024-03-25T18:22:04.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.721+0000 7fe113cd0640 10 monclient: init 2024-03-25T18:22:04.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.722+0000 7fe113cd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:04.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.722+0000 7fe113cd0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:04.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.722+0000 7fe113cd0640 10 monclient(hunting): picked mon.noname-b con 0x7fe10c0a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:22:04.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.722+0000 7fe113cd0640 10 monclient(hunting): picked mon.noname-c con 0x7fe10c13f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:22:04.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.722+0000 7fe113cd0640 10 monclient(hunting): picked mon.noname-a con 0x7fe10c1430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:22:04.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.722+0000 7fe113cd0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c142520 con 0x7fe10c1430c0 2024-03-25T18:22:04.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.722+0000 7fe113cd0640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c0b28a0 con 0x7fe10c13f710 2024-03-25T18:22:04.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.722+0000 7fe113cd0640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c0b2a30 con 0x7fe10c0a98f0 2024-03-25T18:22:04.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.722+0000 7fe113cd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:04.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.722+0000 7fe113cd0640 10 monclient(hunting): authenticate will time out at 7647.988661s 2024-03-25T18:22:04.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.723+0000 7fe11126b640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fe10c13f710 0x7fe10c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:55462/0 (socket says 172.21.15.122:55462) 2024-03-25T18:22:04.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.723+0000 7fe11126b640 1 -- 172.21.15.122:0/2262444817 learned_addr learned my addr 172.21.15.122:0/2262444817 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:22:04.785 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.784+0000 7fe110a6a640 1 -- 172.21.15.122:0/2262444817 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156727657 0 0) 0x7fe10c142520 con 0x7fe10c1430c0 2024-03-25T18:22:04.785 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.784+0000 7fe110a6a640 10 monclient(hunting): _negotiate 2024-03-25T18:22:04.785 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.784+0000 7fe110a6a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:04.785 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.784+0000 7fe110a6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:04.785 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.784+0000 7fe110a6a640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:22:04.785 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.784+0000 7fe110a6a640 1 -- 172.21.15.122:0/2262444817 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f8003520 con 0x7fe10c1430c0 2024-03-25T18:22:04.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.789+0000 7fe110a6a640 1 -- 172.21.15.122:0/2262444817 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3149340726 0 0) 0x7fe0f8003520 con 0x7fe10c1430c0 2024-03-25T18:22:04.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.789+0000 7fe110a6a640 1 -- 172.21.15.122:0/2262444817 >> v1:172.21.15.122:6790/0 conn(0x7fe10c0a98f0 legacy=0x7fe10c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:04.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.789+0000 7fe110a6a640 1 -- 172.21.15.122:0/2262444817 >> v1:172.21.15.204:6789/0 conn(0x7fe10c13f710 legacy=0x7fe10c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:04.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.789+0000 7fe110a6a640 10 monclient: _finish_hunting 0 2024-03-25T18:22:04.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.789+0000 7fe110a6a640 1 monclient: found mon.noname-a 2024-03-25T18:22:04.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.789+0000 7fe110a6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:04.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.789+0000 7fe110a6a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:22:04.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.789+0000 7fe110a6a640 1 -- 172.21.15.122:0/2262444817 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c0b44c0 con 0x7fe10c1430c0 2024-03-25T18:22:04.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.789+0000 7fe110a6a640 10 monclient: _finish_auth 0 2024-03-25T18:22:04.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.789+0000 7fe110a6a640 10 monclient: _check_auth_tickets 2024-03-25T18:22:04.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.789+0000 7fe110a6a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:04.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.789+0000 7fe110a6a640 1 -- 172.21.15.122:0/2262444817 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fe104001a10 con 0x7fe10c1430c0 2024-03-25T18:22:04.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.789+0000 7fe110a6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:04.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.790+0000 7fe110a6a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:04.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.790+0000 7fe110a6a640 10 monclient: dump: 2024-03-25T18:22:04.792 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:04.792 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:04.792 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:04.792 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:04.792 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:04.792 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:04.792 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:04.792 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:04.792 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:04.792 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:04.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.790+0000 7fe113cd0640 5 monclient: authenticate success, global_id 4485 2024-03-25T18:22:04.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.790+0000 7fe113cd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:04.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.805+0000 7fe110a6a640 1 -- 172.21.15.122:0/2262444817 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe104001b70 con 0x7fe10c1430c0 2024-03-25T18:22:04.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.805+0000 7fe110a6a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:04.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.805+0000 7fe110a6a640 1 -- 172.21.15.122:0/2262444817 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fe104001f80 con 0x7fe10c1430c0 2024-03-25T18:22:04.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.805+0000 7fe110a6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:04.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.805+0000 7fe110a6a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:04.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.805+0000 7fe110a6a640 10 monclient: dump: 2024-03-25T18:22:04.806 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:04.806 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:04.806 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:04.806 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:04.806 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:04.806 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:04.807 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:04.807 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:04.807 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:04.807 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:04.807 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.805+0000 7fe113cd0640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:04.807 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.805+0000 7fe113cd0640 4 set_mon_vals no callback set 2024-03-25T18:22:04.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.806+0000 7fe113cd0640 10 monclient: shutdown 2024-03-25T18:22:04.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.806+0000 7fe113cd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:04.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.806+0000 7fe113cd0640 1 -- 172.21.15.122:0/2262444817 >> v1:172.21.15.122:6789/0 conn(0x7fe10c1430c0 legacy=0x7fe10c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:04.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.806+0000 7fe113cd0640 1 -- 172.21.15.122:0/2262444817 shutdown_connections 2024-03-25T18:22:04.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.806+0000 7fe113cd0640 1 -- 172.21.15.122:0/2262444817 >> 172.21.15.122:0/2262444817 conn(0x7fe10c0b7ea0 msgr2=0x7fe10c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:04.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.806+0000 7fe113cd0640 1 -- 172.21.15.122:0/2262444817 shutdown_connections 2024-03-25T18:22:04.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.806+0000 7fe113cd0640 1 -- 172.21.15.122:0/2262444817 wait complete. 2024-03-25T18:22:04.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.807+0000 7fe113cd0640 10 monclient: build_initial_monmap 2024-03-25T18:22:04.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.807+0000 7fe113cd0640 10 monclient: monmap: 2024-03-25T18:22:04.809 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:22:04.809 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:04.809 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:22:04.809 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:22:04.809 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:04.809 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:22:04.809 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:22:04.809 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:22:04.809 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:22:04.809 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:04.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.807+0000 7fe113cd0640 1 librados: starting msgr at 2024-03-25T18:22:04.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.807+0000 7fe113cd0640 1 librados: starting objecter 2024-03-25T18:22:04.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.807+0000 7fe113cd0640 1 Processor -- start 2024-03-25T18:22:04.810 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.807+0000 7fe113cd0640 1 -- start start 2024-03-25T18:22:04.810 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.807+0000 7fe113cd0640 1 librados: setting wanted keys 2024-03-25T18:22:04.810 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.807+0000 7fe113cd0640 1 librados: calling monclient init 2024-03-25T18:22:04.810 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.807+0000 7fe113cd0640 10 monclient: init 2024-03-25T18:22:04.810 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.808+0000 7fe113cd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:04.811 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.808+0000 7fe113cd0640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:04.811 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.808+0000 7fe113cd0640 10 monclient(hunting): picked mon.noname-c con 0x7fe10c0a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:22:04.811 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.808+0000 7fe113cd0640 10 monclient(hunting): picked mon.noname-b con 0x7fe10c13f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:22:04.811 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.808+0000 7fe113cd0640 10 monclient(hunting): picked mon.noname-a con 0x7fe10c1430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:22:04.811 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.808+0000 7fe113cd0640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c0abac0 con 0x7fe10c1430c0 2024-03-25T18:22:04.811 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.808+0000 7fe113cd0640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c0abda0 con 0x7fe10c13f710 2024-03-25T18:22:04.811 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.808+0000 7fe113cd0640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe10c1380e0 con 0x7fe10c0a98f0 2024-03-25T18:22:04.811 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.808+0000 7fe113cd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:04.811 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.808+0000 7fe113cd0640 10 monclient(hunting): authenticate will time out at 7648.074473s 2024-03-25T18:22:04.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.825+0000 7fe11226d640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe10c1430c0 0x7fe10c1e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:46306/0 (socket says 172.21.15.122:46306) 2024-03-25T18:22:04.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.825+0000 7fe11226d640 1 -- 172.21.15.122:0/1619132504 learned_addr learned my addr 172.21.15.122:0/1619132504 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:22:04.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.826+0000 7fe1027fc640 10 client.?.objecter ms_handle_connect 0x7fe10c1430c0 2024-03-25T18:22:04.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.826+0000 7fe1027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:04.837 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.836+0000 7fe1027fc640 1 -- 172.21.15.122:0/1619132504 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970749565 0 0) 0x7fe10c0abac0 con 0x7fe10c1430c0 2024-03-25T18:22:04.870 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.836+0000 7fe1027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:04.871 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.836+0000 7fe1027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:04.871 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.836+0000 7fe1027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:04.871 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.836+0000 7fe1027fc640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:22:04.871 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.836+0000 7fe1027fc640 1 -- 172.21.15.122:0/1619132504 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0f0004eb0 con 0x7fe10c1430c0 2024-03-25T18:22:04.872 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.840+0000 7fe1027fc640 10 client.?.objecter ms_handle_connect 0x7fe10c0a98f0 2024-03-25T18:22:04.872 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.840+0000 7fe1027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:04.872 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.851+0000 7fe1027fc640 1 -- 172.21.15.122:0/1619132504 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3399709906 0 0) 0x7fe0f0004eb0 con 0x7fe10c1430c0 2024-03-25T18:22:04.872 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.851+0000 7fe1027fc640 1 -- 172.21.15.122:0/1619132504 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe10c0abac0 con 0x7fe10c1430c0 2024-03-25T18:22:04.872 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.851+0000 7fe1027fc640 1 -- 172.21.15.122:0/1619132504 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fe104001fc0 con 0x7fe10c1430c0 2024-03-25T18:22:04.872 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.851+0000 7fe1027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:04.872 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.851+0000 7fe1027fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:04.872 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.851+0000 7fe1027fc640 10 monclient(hunting): dump: 2024-03-25T18:22:04.872 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:04.873 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:04.873 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:04.873 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:04.873 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:04.873 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:04.873 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:04.873 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:04.873 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:04.873 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:04.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.851+0000 7fe1027fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:04.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.854+0000 7fe1027fc640 10 client.?.objecter ms_handle_connect 0x7fe10c13f710 2024-03-25T18:22:04.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.854+0000 7fe1027fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:04.877 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe1027fc640 1 -- 172.21.15.122:0/1619132504 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3718876265 0 0) 0x7fe10c0abac0 con 0x7fe10c1430c0 2024-03-25T18:22:04.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe1027fc640 1 -- 172.21.15.122:0/1619132504 >> v1:172.21.15.122:6790/0 conn(0x7fe10c0a98f0 legacy=0x7fe10c0ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:04.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe1027fc640 1 -- 172.21.15.122:0/1619132504 >> v1:172.21.15.204:6789/0 conn(0x7fe10c13f710 legacy=0x7fe10c138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:04.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe1027fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:04.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe1027fc640 1 monclient: found mon.a 2024-03-25T18:22:04.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe1027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:04.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe1027fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:22:04.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe1027fc640 1 -- 172.21.15.122:0/1619132504 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe10c137d90 con 0x7fe10c1430c0 2024-03-25T18:22:04.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe1027fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:04.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe1027fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:04.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe1027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:04.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe113cd0640 5 monclient: authenticate success, global_id 4488 2024-03-25T18:22:04.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe113cd0640 10 monclient: _renew_subs 2024-03-25T18:22:04.879 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe113cd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:22:04.879 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe113cd0640 1 -- 172.21.15.122:0/1619132504 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe10c139240 con 0x7fe10c1430c0 2024-03-25T18:22:04.879 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe113cd0640 10 client.4488.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:04.879 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe113cd0640 10 monclient: _renew_subs 2024-03-25T18:22:04.879 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe113cd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:22:04.879 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe113cd0640 1 -- 172.21.15.122:0/1619132504 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe10c136830 con 0x7fe10c1430c0 2024-03-25T18:22:04.879 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.876+0000 7fe113cd0640 1 librados: init done 2024-03-25T18:22:04.879 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.877+0000 7fe113cd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:04.880 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.877+0000 7fe113cd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:04.880 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.877+0000 7fe113cd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:22:04.880 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:04.877+0000 7fe113cd0640 1 -- 172.21.15.122:0/1619132504 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe10c0627b0 con 0x7fe10c1430c0 2024-03-25T18:22:05.223 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.222+0000 7fe1027fc640 1 -- 172.21.15.122:0/1619132504 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe104003c20 con 0x7fe10c1430c0 2024-03-25T18:22:05.224 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.222+0000 7fe1027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:05.224 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.222+0000 7fe1027fc640 1 -- 172.21.15.122:0/1619132504 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fe104003ff0 con 0x7fe10c1430c0 2024-03-25T18:22:05.224 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.222+0000 7fe1027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:05.224 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.222+0000 7fe1027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:05.224 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.222+0000 7fe1027fc640 10 monclient: dump: 2024-03-25T18:22:05.224 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:05.224 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:05.224 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:05.224 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:05.224 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:05.224 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:05.224 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:05.224 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:05.225 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:05.225 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:05.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.222+0000 7fe103fff640 4 set_mon_vals no callback set 2024-03-25T18:22:05.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.223+0000 7fe1027fc640 1 -- 172.21.15.122:0/1619132504 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fe104004210 con 0x7fe10c1430c0 2024-03-25T18:22:05.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.224+0000 7fe111a6c640 10 monclient: get_auth_request con 0x7fe0f006dc70 auth_method 0 2024-03-25T18:22:05.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.224+0000 7fe1027fc640 10 client.4488.objecter ms_handle_connect 0x7fe0f006dc70 2024-03-25T18:22:05.232 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.230+0000 7fe1027fc640 1 -- 172.21.15.122:0/1619132504 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fe104054c80 con 0x7fe10c1430c0 2024-03-25T18:22:05.232 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.230+0000 7fe1027fc640 10 client.4488.objecter ms_dispatch 0x7fe10c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:05.232 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.230+0000 7fe1027fc640 3 client.4488.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:05.232 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.230+0000 7fe1027fc640 3 client.4488.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:05.232 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.230+0000 7fe1027fc640 20 client.4488.objecter dump_active .. 0 homeless 2024-03-25T18:22:05.232 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.230+0000 7fe1027fc640 1 -- 172.21.15.122:0/1619132504 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe104050790 con 0x7fe10c1430c0 2024-03-25T18:22:05.232 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.230+0000 7fe1027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:05.232 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.230+0000 7fe1027fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:05.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.482+0000 7fe113cd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:05.484 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.482+0000 7fe113cd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:05.484 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.482+0000 7fe113cd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:22:05.484 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.482+0000 7fe113cd0640 1 -- 172.21.15.122:0/1619132504 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe10c13f070 con 0x7fe10c1430c0 2024-03-25T18:22:05.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.507+0000 7fe1027fc640 1 -- 172.21.15.122:0/1619132504 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fe1040526c0 con 0x7fe10c1430c0 2024-03-25T18:22:05.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.507+0000 7fe1027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:05.508 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.507+0000 7fe1027fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:05.509 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:22:05.509 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:05.511 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.509+0000 7fe113cd0640 10 librados: watch_flush enter 2024-03-25T18:22:05.511 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.509+0000 7fe113cd0640 10 librados: watch_flush exit 2024-03-25T18:22:05.511 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.509+0000 7fe113cd0640 20 client.4488.objecter shutdown clearing up homeless session... 2024-03-25T18:22:05.511 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.509+0000 7fe113cd0640 10 client.4488.objecter successfully canceled tick 2024-03-25T18:22:05.511 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.509+0000 7fe113cd0640 1 -- 172.21.15.122:0/1619132504 >> v1:172.21.15.204:6800/676228005 conn(0x7fe0f006dc70 legacy=0x7fe0f0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:05.511 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.509+0000 7fe113cd0640 10 monclient: shutdown 2024-03-25T18:22:05.511 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.509+0000 7fe113cd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:05.511 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.509+0000 7fe113cd0640 1 -- 172.21.15.122:0/1619132504 >> v1:172.21.15.122:6789/0 conn(0x7fe10c1430c0 legacy=0x7fe10c1e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:05.511 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.509+0000 7fe113cd0640 1 -- 172.21.15.122:0/1619132504 shutdown_connections 2024-03-25T18:22:05.511 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.509+0000 7fe113cd0640 1 -- 172.21.15.122:0/1619132504 >> 172.21.15.122:0/1619132504 conn(0x7fe10c0b7ea0 msgr2=0x7fe10c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:05.512 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.509+0000 7fe113cd0640 1 -- 172.21.15.122:0/1619132504 shutdown_connections 2024-03-25T18:22:05.512 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.509+0000 7fe113cd0640 1 -- 172.21.15.122:0/1619132504 wait complete. 2024-03-25T18:22:05.512 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:05.509+0000 7fe113cd0640 1 librados: shutdown 2024-03-25T18:22:05.521 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:06.495 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:06:00.238 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:06.505 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:06:00.248 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:06.531 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:06:00.273 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:06.552 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:06:00.298 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:06.558 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:06:00.304 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:06.559 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:06:00.301 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:06.571 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:06:00.314 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:06.597 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:06:00.342 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:06.599 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:06:00.341 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:12.328 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:12.441 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.439+0000 7f676e1d6640 10 monclient: get_monmap_and_config 2024-03-25T18:22:12.441 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.439+0000 7f676e1d6640 10 monclient: build_initial_monmap 2024-03-25T18:22:12.441 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.439+0000 7f676e1d6640 10 monclient: monmap: 2024-03-25T18:22:12.441 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:22:12.441 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:12.441 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:22:12.440866+0000 2024-03-25T18:22:12.441 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:22:12.440866+0000 2024-03-25T18:22:12.441 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:12.441 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:12.441 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:22:12.441 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:22:12.441 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:22:12.441 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:12.443 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.440+0000 7f676e1d6640 1 Processor -- start 2024-03-25T18:22:12.443 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.440+0000 7f676e1d6640 1 -- start start 2024-03-25T18:22:12.443 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.440+0000 7f676e1d6640 10 monclient: init 2024-03-25T18:22:12.443 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.441+0000 7f676e1d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:12.443 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.441+0000 7f676e1d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:12.443 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.441+0000 7f676e1d6640 10 monclient(hunting): picked mon.noname-a con 0x7f67680a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:22:12.443 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.441+0000 7f676e1d6640 10 monclient(hunting): picked mon.noname-c con 0x7f676813f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:22:12.443 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.441+0000 7f676e1d6640 10 monclient(hunting): picked mon.noname-b con 0x7f67681430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:22:12.443 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.441+0000 7f676e1d6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6768142520 con 0x7f67680a98f0 2024-03-25T18:22:12.443 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.441+0000 7f676e1d6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67680b28a0 con 0x7f676813f710 2024-03-25T18:22:12.443 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.441+0000 7f676e1d6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67680b2a30 con 0x7f67681430c0 2024-03-25T18:22:12.444 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.441+0000 7f676e1d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:12.444 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.441+0000 7f676e1d6640 10 monclient(hunting): authenticate will time out at 7655.707618s 2024-03-25T18:22:12.451 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.450+0000 7f6766ffd640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f676813f710 0x7f67681429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:48532/0 (socket says 172.21.15.122:48532) 2024-03-25T18:22:12.452 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.450+0000 7f6766ffd640 1 -- 172.21.15.122:0/1288914112 learned_addr learned my addr 172.21.15.122:0/1288914112 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:22:12.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.459+0000 7f67667fc640 1 -- 172.21.15.122:0/1288914112 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2929053675 0 0) 0x7f67680b28a0 con 0x7f676813f710 2024-03-25T18:22:12.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.459+0000 7f67667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:12.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.459+0000 7f67667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:12.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.459+0000 7f67667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:12.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.459+0000 7f67667fc640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:22:12.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.459+0000 7f67667fc640 1 -- 172.21.15.122:0/1288914112 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6750003520 con 0x7f676813f710 2024-03-25T18:22:12.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.463+0000 7f67667fc640 1 -- 172.21.15.122:0/1288914112 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2271957492 0 0) 0x7f6750003520 con 0x7f676813f710 2024-03-25T18:22:12.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.463+0000 7f67667fc640 1 -- 172.21.15.122:0/1288914112 >> v1:172.21.15.122:6790/0 conn(0x7f67681430c0 legacy=0x7f676814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:22:12.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.463+0000 7f67667fc640 1 -- 172.21.15.122:0/1288914112 >> v1:172.21.15.122:6789/0 conn(0x7f67680a98f0 legacy=0x7f67680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:22:12.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.463+0000 7f67667fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:12.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.463+0000 7f67667fc640 1 monclient: found mon.noname-c 2024-03-25T18:22:12.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.463+0000 7f67667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:12.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.463+0000 7f67667fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:22:12.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.463+0000 7f67667fc640 1 -- 172.21.15.122:0/1288914112 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67680b44c0 con 0x7f676813f710 2024-03-25T18:22:12.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.463+0000 7f67667fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:12.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.463+0000 7f67667fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:12.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.463+0000 7f67667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:12.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.463+0000 7f67667fc640 1 -- 172.21.15.122:0/1288914112 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f675c001690 con 0x7f676813f710 2024-03-25T18:22:12.466 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.463+0000 7f67667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:12.466 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.463+0000 7f67667fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:12.466 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.463+0000 7f67667fc640 10 monclient: dump: 2024-03-25T18:22:12.466 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:12.466 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:12.466 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:12.466 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:12.466 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:12.466 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:12.466 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:12.466 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:12.466 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:12.466 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:12.466 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.463+0000 7f676e1d6640 5 monclient: authenticate success, global_id 4483 2024-03-25T18:22:12.466 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.463+0000 7f676e1d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:12.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.475+0000 7f67667fc640 1 -- 172.21.15.122:0/1288914112 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f675c001860 con 0x7f676813f710 2024-03-25T18:22:12.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.475+0000 7f67667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:12.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.475+0000 7f67667fc640 1 -- 172.21.15.122:0/1288914112 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f675c001cb0 con 0x7f676813f710 2024-03-25T18:22:12.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.475+0000 7f67667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:12.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.475+0000 7f67667fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:12.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.475+0000 7f67667fc640 10 monclient: dump: 2024-03-25T18:22:12.477 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:12.477 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:12.477 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:12.477 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:12.477 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:12.477 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:12.477 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:12.478 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:12.478 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:12.478 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:12.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.475+0000 7f676e1d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:12.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.475+0000 7f676e1d6640 4 set_mon_vals no callback set 2024-03-25T18:22:12.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.476+0000 7f676e1d6640 10 monclient: shutdown 2024-03-25T18:22:12.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.476+0000 7f676e1d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:12.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.476+0000 7f676e1d6640 1 -- 172.21.15.122:0/1288914112 >> v1:172.21.15.204:6789/0 conn(0x7f676813f710 legacy=0x7f67681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:12.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.476+0000 7f676e1d6640 1 -- 172.21.15.122:0/1288914112 shutdown_connections 2024-03-25T18:22:12.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.476+0000 7f676e1d6640 1 -- 172.21.15.122:0/1288914112 >> 172.21.15.122:0/1288914112 conn(0x7f67680b7ea0 msgr2=0x7f67680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:12.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.476+0000 7f676e1d6640 1 -- 172.21.15.122:0/1288914112 shutdown_connections 2024-03-25T18:22:12.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.476+0000 7f676e1d6640 1 -- 172.21.15.122:0/1288914112 wait complete. 2024-03-25T18:22:12.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.477+0000 7f676e1d6640 10 monclient: build_initial_monmap 2024-03-25T18:22:12.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.477+0000 7f676e1d6640 10 monclient: monmap: 2024-03-25T18:22:12.480 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:22:12.480 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:12.480 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:22:12.480 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:22:12.480 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:12.480 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:22:12.480 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:22:12.480 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:22:12.480 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:22:12.480 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:12.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.478+0000 7f676e1d6640 1 librados: starting msgr at 2024-03-25T18:22:12.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.478+0000 7f676e1d6640 1 librados: starting objecter 2024-03-25T18:22:12.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.478+0000 7f676e1d6640 1 Processor -- start 2024-03-25T18:22:12.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.478+0000 7f676e1d6640 1 -- start start 2024-03-25T18:22:12.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.478+0000 7f676e1d6640 1 librados: setting wanted keys 2024-03-25T18:22:12.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.478+0000 7f676e1d6640 1 librados: calling monclient init 2024-03-25T18:22:12.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.478+0000 7f676e1d6640 10 monclient: init 2024-03-25T18:22:12.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.479+0000 7f676e1d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:12.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.479+0000 7f676e1d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:12.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.479+0000 7f676e1d6640 10 monclient(hunting): picked mon.noname-a con 0x7f67680a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:22:12.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.479+0000 7f676e1d6640 10 monclient(hunting): picked mon.noname-b con 0x7f676813f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:22:12.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.479+0000 7f676e1d6640 10 monclient(hunting): picked mon.noname-c con 0x7f67681430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:22:12.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.479+0000 7f676e1d6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67680b0d00 con 0x7f67680a98f0 2024-03-25T18:22:12.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.479+0000 7f676e1d6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67681d82b0 con 0x7f676813f710 2024-03-25T18:22:12.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.479+0000 7f676e1d6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f676814e0c0 con 0x7f67681430c0 2024-03-25T18:22:12.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.479+0000 7f676e1d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:12.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.479+0000 7f676e1d6640 10 monclient(hunting): authenticate will time out at 7655.745802s 2024-03-25T18:22:12.490 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.489+0000 7f6767fff640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f67681430c0 0x7f67681d7ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:35802/0 (socket says 172.21.15.122:35802) 2024-03-25T18:22:12.490 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.489+0000 7f6767fff640 1 -- 172.21.15.122:0/121581838 learned_addr learned my addr 172.21.15.122:0/121581838 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:22:12.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.491+0000 7f674ffff640 10 client.?.objecter ms_handle_connect 0x7f67681430c0 2024-03-25T18:22:12.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.491+0000 7f674ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:12.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.504+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095368758 0 0) 0x7f676814e0c0 con 0x7f67681430c0 2024-03-25T18:22:12.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.504+0000 7f674ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:12.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.504+0000 7f674ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:12.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.504+0000 7f674ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:12.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.504+0000 7f674ffff640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:22:12.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.504+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6754004eb0 con 0x7f67681430c0 2024-03-25T18:22:12.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.508+0000 7f674ffff640 10 client.?.objecter ms_handle_connect 0x7f67680a98f0 2024-03-25T18:22:12.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.508+0000 7f674ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:12.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.518+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3059689365 0 0) 0x7f67680b0d00 con 0x7f67680a98f0 2024-03-25T18:22:12.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.518+0000 7f674ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:12.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.518+0000 7f674ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:12.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.518+0000 7f674ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:12.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.518+0000 7f674ffff640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:22:12.519 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.518+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f676814e0c0 con 0x7f67680a98f0 2024-03-25T18:22:12.524 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.522+0000 7f674ffff640 10 client.?.objecter ms_handle_connect 0x7f676813f710 2024-03-25T18:22:12.524 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.522+0000 7f674ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:12.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.524+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 451649789 0 0) 0x7f6754004eb0 con 0x7f67681430c0 2024-03-25T18:22:12.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.524+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67680b0d00 con 0x7f67681430c0 2024-03-25T18:22:12.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.524+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f67600042e0 con 0x7f67681430c0 2024-03-25T18:22:12.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.524+0000 7f674ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:12.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.524+0000 7f674ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:12.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.524+0000 7f674ffff640 10 monclient(hunting): dump: 2024-03-25T18:22:12.526 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:12.526 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:12.526 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:12.526 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:12.526 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:12.526 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:12.526 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:12.526 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:12.526 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:12.526 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:12.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.524+0000 7f674ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:12.534 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.532+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 785746992 0 0) 0x7f67681d82b0 con 0x7f676813f710 2024-03-25T18:22:12.534 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.532+0000 7f674ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:12.534 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.532+0000 7f674ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:12.534 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.532+0000 7f674ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:12.534 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.532+0000 7f674ffff640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:22:12.534 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.532+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6754004eb0 con 0x7f676813f710 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1226507844 0 0) 0x7f67680b0d00 con 0x7f67681430c0 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 >> v1:172.21.15.204:6789/0 conn(0x7f676813f710 legacy=0x7f67680af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 >> v1:172.21.15.122:6789/0 conn(0x7f67680a98f0 legacy=0x7f67681d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f674ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f674ffff640 1 monclient: found mon.c 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f674ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f674ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676814e250 con 0x7f67681430c0 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f674ffff640 10 monclient: _finish_auth 0 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f674ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f674ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f676e1d6640 5 monclient: authenticate success, global_id 4457 2024-03-25T18:22:12.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f676e1d6640 10 monclient: _renew_subs 2024-03-25T18:22:12.537 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f676e1d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:22:12.537 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f676e1d6640 1 -- 172.21.15.122:0/121581838 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67681dac00 con 0x7f67681430c0 2024-03-25T18:22:12.537 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f676e1d6640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:12.537 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f676e1d6640 10 monclient: _renew_subs 2024-03-25T18:22:12.537 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f676e1d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:22:12.537 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f676e1d6640 1 -- 172.21.15.122:0/121581838 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67680b1290 con 0x7f67681430c0 2024-03-25T18:22:12.537 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.533+0000 7f676e1d6640 1 librados: init done 2024-03-25T18:22:12.537 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.534+0000 7f676e1d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:12.537 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.534+0000 7f676e1d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:12.537 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.534+0000 7f676e1d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:22:12.537 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.534+0000 7f676e1d6640 1 -- 172.21.15.122:0/121581838 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67680b1590 con 0x7f67681430c0 2024-03-25T18:22:12.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.832+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67600047f0 con 0x7f67681430c0 2024-03-25T18:22:12.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.832+0000 7f674ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:12.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.832+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f67600013f0 con 0x7f67681430c0 2024-03-25T18:22:12.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.832+0000 7f674ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:12.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.832+0000 7f674ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:12.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.832+0000 7f674ffff640 10 monclient: dump: 2024-03-25T18:22:12.834 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:12.834 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:12.834 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:12.834 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:12.834 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:12.834 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:12.834 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:12.834 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:12.834 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:12.834 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:12.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.832+0000 7f67657fa640 4 set_mon_vals no callback set 2024-03-25T18:22:12.835 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.833+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f67600015d0 con 0x7f67681430c0 2024-03-25T18:22:12.835 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.833+0000 7f67677fe640 10 monclient: get_auth_request con 0x7f675406ddb0 auth_method 0 2024-03-25T18:22:12.835 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.834+0000 7f674ffff640 10 client.4457.objecter ms_handle_connect 0x7f675406ddb0 2024-03-25T18:22:12.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.839+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f6760054ac0 con 0x7f67681430c0 2024-03-25T18:22:12.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.839+0000 7f674ffff640 10 client.4457.objecter ms_dispatch 0x7f6768000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:12.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.839+0000 7f674ffff640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:12.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.839+0000 7f674ffff640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:12.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.839+0000 7f674ffff640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-25T18:22:12.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.839+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6760050400 con 0x7f67681430c0 2024-03-25T18:22:12.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.840+0000 7f674ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:12.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:12.840+0000 7f674ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:13.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.054+0000 7f676e1d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:13.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.054+0000 7f676e1d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:13.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.054+0000 7f676e1d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:22:13.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.054+0000 7f676e1d6640 1 -- 172.21.15.122:0/121581838 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67681db290 con 0x7f67681430c0 2024-03-25T18:22:13.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.296+0000 7f674ffff640 1 -- 172.21.15.122:0/121581838 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f6760052500 con 0x7f67681430c0 2024-03-25T18:22:13.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.297+0000 7f674ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:13.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.297+0000 7f674ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:13.298 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:22:13.298 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:13.301 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.299+0000 7f676e1d6640 10 librados: watch_flush enter 2024-03-25T18:22:13.301 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.299+0000 7f676e1d6640 10 librados: watch_flush exit 2024-03-25T18:22:13.301 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.299+0000 7f676e1d6640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-25T18:22:13.301 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.299+0000 7f676e1d6640 10 client.4457.objecter successfully canceled tick 2024-03-25T18:22:13.301 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.299+0000 7f676e1d6640 1 -- 172.21.15.122:0/121581838 >> v1:172.21.15.204:6800/676228005 conn(0x7f675406ddb0 legacy=0x7f6754070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:13.301 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.299+0000 7f676e1d6640 10 monclient: shutdown 2024-03-25T18:22:13.301 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.299+0000 7f676e1d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:13.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.299+0000 7f676e1d6640 1 -- 172.21.15.122:0/121581838 >> v1:172.21.15.122:6790/0 conn(0x7f67681430c0 legacy=0x7f67681d7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:13.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.299+0000 7f676e1d6640 1 -- 172.21.15.122:0/121581838 shutdown_connections 2024-03-25T18:22:13.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.299+0000 7f676e1d6640 1 -- 172.21.15.122:0/121581838 >> 172.21.15.122:0/121581838 conn(0x7f67680b7ea0 msgr2=0x7f6768074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:13.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.299+0000 7f676e1d6640 1 -- 172.21.15.122:0/121581838 shutdown_connections 2024-03-25T18:22:13.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.299+0000 7f676e1d6640 1 -- 172.21.15.122:0/121581838 wait complete. 2024-03-25T18:22:13.302 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:13.300+0000 7f676e1d6640 1 librados: shutdown 2024-03-25T18:22:13.312 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:20.117 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:20.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.234+0000 7f16bff6b640 10 monclient: get_monmap_and_config 2024-03-25T18:22:20.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.234+0000 7f16bff6b640 10 monclient: build_initial_monmap 2024-03-25T18:22:20.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.234+0000 7f16bff6b640 10 monclient: monmap: 2024-03-25T18:22:20.235 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:22:20.236 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:20.236 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:22:20.235546+0000 2024-03-25T18:22:20.236 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:22:20.235546+0000 2024-03-25T18:22:20.236 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:20.236 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:20.236 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:22:20.236 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:22:20.236 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:22:20.236 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:20.236 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.235+0000 7f16bff6b640 1 Processor -- start 2024-03-25T18:22:20.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.235+0000 7f16bff6b640 1 -- start start 2024-03-25T18:22:20.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.235+0000 7f16bff6b640 10 monclient: init 2024-03-25T18:22:20.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.236+0000 7f16bff6b640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:20.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.236+0000 7f16bff6b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:20.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.236+0000 7f16bff6b640 10 monclient(hunting): picked mon.noname-b con 0x7f16b80705f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:22:20.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.236+0000 7f16bff6b640 10 monclient(hunting): picked mon.noname-c con 0x7f16b80709d0 addr v1:172.21.15.204:6789/0 2024-03-25T18:22:20.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.236+0000 7f16bff6b640 10 monclient(hunting): picked mon.noname-a con 0x7f16b8140a40 addr v1:172.21.15.122:6789/0 2024-03-25T18:22:20.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.236+0000 7f16bff6b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b8071980 con 0x7f16b8140a40 2024-03-25T18:22:20.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.236+0000 7f16bff6b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b80728b0 con 0x7f16b80709d0 2024-03-25T18:22:20.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.236+0000 7f16bff6b640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b813da10 con 0x7f16b80705f0 2024-03-25T18:22:20.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.236+0000 7f16bff6b640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:20.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.236+0000 7f16bff6b640 10 monclient(hunting): authenticate will time out at 7663.502839s 2024-03-25T18:22:20.243 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.242+0000 7f16bdd07640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f16b80705f0 0x7f16b813d300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:55324/0 (socket says 172.21.15.122:55324) 2024-03-25T18:22:20.243 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.242+0000 7f16bdd07640 1 -- 172.21.15.122:0/3146782993 learned_addr learned my addr 172.21.15.122:0/3146782993 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:22:20.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.252+0000 7f16bcd05640 1 -- 172.21.15.122:0/3146782993 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2790876401 0 0) 0x7f16b813da10 con 0x7f16b80705f0 2024-03-25T18:22:20.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.252+0000 7f16bcd05640 10 monclient(hunting): _negotiate 2024-03-25T18:22:20.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.252+0000 7f16bcd05640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:20.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.252+0000 7f16bcd05640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:20.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.252+0000 7f16bcd05640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:22:20.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.252+0000 7f16bcd05640 1 -- 172.21.15.122:0/3146782993 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a4003520 con 0x7f16b80705f0 2024-03-25T18:22:20.270 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.268+0000 7f16bcd05640 1 -- 172.21.15.122:0/3146782993 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1339383646 0 0) 0x7f16a4003520 con 0x7f16b80705f0 2024-03-25T18:22:20.270 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.268+0000 7f16bcd05640 1 -- 172.21.15.122:0/3146782993 >> v1:172.21.15.204:6789/0 conn(0x7f16b80709d0 legacy=0x7f16b8140330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:20.270 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.268+0000 7f16bcd05640 1 -- 172.21.15.122:0/3146782993 >> v1:172.21.15.122:6789/0 conn(0x7f16b8140a40 legacy=0x7f16b814ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:22:20.270 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.268+0000 7f16bcd05640 10 monclient: _finish_hunting 0 2024-03-25T18:22:20.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.268+0000 7f16bcd05640 1 monclient: found mon.noname-b 2024-03-25T18:22:20.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.269+0000 7f16bcd05640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:20.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.269+0000 7f16bcd05640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:22:20.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.269+0000 7f16bcd05640 1 -- 172.21.15.122:0/3146782993 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b80b0230 con 0x7f16b80705f0 2024-03-25T18:22:20.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.269+0000 7f16bcd05640 10 monclient: _finish_auth 0 2024-03-25T18:22:20.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.269+0000 7f16bcd05640 10 monclient: _check_auth_tickets 2024-03-25T18:22:20.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.269+0000 7f16bcd05640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:20.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.269+0000 7f16bcd05640 1 -- 172.21.15.122:0/3146782993 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f16b4001690 con 0x7f16b80705f0 2024-03-25T18:22:20.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.269+0000 7f16bff6b640 5 monclient: authenticate success, global_id 4460 2024-03-25T18:22:20.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.269+0000 7f16bff6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:20.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.269+0000 7f16bcd05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:20.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.269+0000 7f16bcd05640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:20.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.269+0000 7f16bcd05640 10 monclient: dump: 2024-03-25T18:22:20.271 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:20.271 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:20.271 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:20.271 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:20.272 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:20.272 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:20.272 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:20.272 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:20.272 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:20.272 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:20.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.269+0000 7f16bff6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:20.287 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.286+0000 7f16bcd05640 1 -- 172.21.15.122:0/3146782993 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16b4001860 con 0x7f16b80705f0 2024-03-25T18:22:20.287 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.286+0000 7f16bcd05640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:20.287 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.286+0000 7f16bcd05640 1 -- 172.21.15.122:0/3146782993 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f16b4001cb0 con 0x7f16b80705f0 2024-03-25T18:22:20.287 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.286+0000 7f16bff6b640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:20.287 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.286+0000 7f16bff6b640 4 set_mon_vals no callback set 2024-03-25T18:22:20.287 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.286+0000 7f16bcd05640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:20.287 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.286+0000 7f16bcd05640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:20.287 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.286+0000 7f16bcd05640 10 monclient: dump: 2024-03-25T18:22:20.287 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:20.287 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:20.287 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:20.287 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:20.287 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:20.287 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:20.288 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:20.288 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:20.288 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:20.288 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:20.288 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.286+0000 7f16bff6b640 10 monclient: shutdown 2024-03-25T18:22:20.288 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.286+0000 7f16bff6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:20.288 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.286+0000 7f16bff6b640 1 -- 172.21.15.122:0/3146782993 >> v1:172.21.15.122:6790/0 conn(0x7f16b80705f0 legacy=0x7f16b813d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:20.288 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.287+0000 7f16bff6b640 1 -- 172.21.15.122:0/3146782993 shutdown_connections 2024-03-25T18:22:20.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.287+0000 7f16bff6b640 1 -- 172.21.15.122:0/3146782993 >> 172.21.15.122:0/3146782993 conn(0x7f16b80b5c00 msgr2=0x7f16b80b6010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:20.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.287+0000 7f16bff6b640 1 -- 172.21.15.122:0/3146782993 shutdown_connections 2024-03-25T18:22:20.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.287+0000 7f16bff6b640 1 -- 172.21.15.122:0/3146782993 wait complete. 2024-03-25T18:22:20.290 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.288+0000 7f16bff6b640 10 monclient: build_initial_monmap 2024-03-25T18:22:20.290 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.288+0000 7f16bff6b640 10 monclient: monmap: 2024-03-25T18:22:20.291 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:22:20.291 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:20.291 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:22:20.291 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:22:20.291 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:20.291 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:22:20.291 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:22:20.291 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:22:20.291 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:22:20.291 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:20.291 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.288+0000 7f16bff6b640 1 librados: starting msgr at 2024-03-25T18:22:20.291 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.288+0000 7f16bff6b640 1 librados: starting objecter 2024-03-25T18:22:20.291 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.288+0000 7f16bff6b640 1 Processor -- start 2024-03-25T18:22:20.291 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.288+0000 7f16bff6b640 1 -- start start 2024-03-25T18:22:20.291 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.288+0000 7f16bff6b640 1 librados: setting wanted keys 2024-03-25T18:22:20.291 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.288+0000 7f16bff6b640 1 librados: calling monclient init 2024-03-25T18:22:20.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.288+0000 7f16bff6b640 10 monclient: init 2024-03-25T18:22:20.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.289+0000 7f16bff6b640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:20.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.289+0000 7f16bff6b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:20.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.289+0000 7f16bff6b640 10 monclient(hunting): picked mon.noname-c con 0x7f16b80705f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:22:20.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.289+0000 7f16bff6b640 10 monclient(hunting): picked mon.noname-b con 0x7f16b80709d0 addr v1:172.21.15.204:6789/0 2024-03-25T18:22:20.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.289+0000 7f16bff6b640 10 monclient(hunting): picked mon.noname-a con 0x7f16b8140a40 addr v1:172.21.15.122:6789/0 2024-03-25T18:22:20.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.289+0000 7f16bff6b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b80aad90 con 0x7f16b8140a40 2024-03-25T18:22:20.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.289+0000 7f16bff6b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b80a96e0 con 0x7f16b80709d0 2024-03-25T18:22:20.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.289+0000 7f16bff6b640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b80a99c0 con 0x7f16b80705f0 2024-03-25T18:22:20.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.289+0000 7f16bff6b640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:20.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.289+0000 7f16bff6b640 10 monclient(hunting): authenticate will time out at 7663.555464s 2024-03-25T18:22:20.297 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.296+0000 7f16be508640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f16b8140a40 0x7f16b81e2ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:32950/0 (socket says 172.21.15.122:32950) 2024-03-25T18:22:20.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.296+0000 7f16be508640 1 -- 172.21.15.122:0/2226511107 learned_addr learned my addr 172.21.15.122:0/2226511107 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:22:20.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.297+0000 7f16ae7fc640 10 client.?.objecter ms_handle_connect 0x7f16b8140a40 2024-03-25T18:22:20.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.297+0000 7f16ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:20.311 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.310+0000 7f16ae7fc640 10 client.?.objecter ms_handle_connect 0x7f16b80705f0 2024-03-25T18:22:20.311 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.310+0000 7f16ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:20.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.312+0000 7f16ae7fc640 1 -- 172.21.15.122:0/2226511107 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2458630759 0 0) 0x7f16b80aad90 con 0x7f16b8140a40 2024-03-25T18:22:20.314 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.312+0000 7f16ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:20.314 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.312+0000 7f16ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:20.314 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.312+0000 7f16ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:20.314 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.312+0000 7f16ae7fc640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:22:20.314 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.312+0000 7f16ae7fc640 1 -- 172.21.15.122:0/2226511107 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f169c004eb0 con 0x7f16b8140a40 2024-03-25T18:22:20.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.317+0000 7f16ae7fc640 10 client.?.objecter ms_handle_connect 0x7f16b80709d0 2024-03-25T18:22:20.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.317+0000 7f16ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:20.332 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.331+0000 7f16ae7fc640 1 -- 172.21.15.122:0/2226511107 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3585727146 0 0) 0x7f169c004eb0 con 0x7f16b8140a40 2024-03-25T18:22:20.332 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.331+0000 7f16ae7fc640 1 -- 172.21.15.122:0/2226511107 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16b80aad90 con 0x7f16b8140a40 2024-03-25T18:22:20.332 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.331+0000 7f16ae7fc640 1 -- 172.21.15.122:0/2226511107 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f16b0001ad0 con 0x7f16b8140a40 2024-03-25T18:22:20.332 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.331+0000 7f16ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:20.332 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.331+0000 7f16ae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:20.332 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.331+0000 7f16ae7fc640 10 monclient(hunting): dump: 2024-03-25T18:22:20.333 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:20.333 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:20.333 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:20.333 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:20.333 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:20.333 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:20.333 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:20.333 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:20.333 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:20.333 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:20.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.331+0000 7f16ae7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:20.343 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.341+0000 7f16ae7fc640 1 -- 172.21.15.122:0/2226511107 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2977846735 0 0) 0x7f16b80aad90 con 0x7f16b8140a40 2024-03-25T18:22:20.343 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16ae7fc640 1 -- 172.21.15.122:0/2226511107 >> v1:172.21.15.122:6790/0 conn(0x7f16b80705f0 legacy=0x7f16b8133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:20.343 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16ae7fc640 1 -- 172.21.15.122:0/2226511107 >> v1:172.21.15.204:6789/0 conn(0x7f16b80709d0 legacy=0x7f16b81d7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:20.343 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16ae7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:20.343 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16ae7fc640 1 monclient: found mon.a 2024-03-25T18:22:20.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:20.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16ae7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:22:20.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16ae7fc640 1 -- 172.21.15.122:0/2226511107 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b806d160 con 0x7f16b8140a40 2024-03-25T18:22:20.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16ae7fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:20.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16ae7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:20.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:20.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16bff6b640 5 monclient: authenticate success, global_id 4494 2024-03-25T18:22:20.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16bff6b640 10 monclient: _renew_subs 2024-03-25T18:22:20.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16bff6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:22:20.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16bff6b640 1 -- 172.21.15.122:0/2226511107 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b806d9d0 con 0x7f16b8140a40 2024-03-25T18:22:20.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16bff6b640 10 client.4494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:20.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16bff6b640 10 monclient: _renew_subs 2024-03-25T18:22:20.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16bff6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:22:20.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16bff6b640 1 -- 172.21.15.122:0/2226511107 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b8144100 con 0x7f16b8140a40 2024-03-25T18:22:20.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.342+0000 7f16bff6b640 1 librados: init done 2024-03-25T18:22:20.345 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.343+0000 7f16bff6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:20.345 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.343+0000 7f16bff6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:20.345 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.343+0000 7f16bff6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:22:20.345 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.343+0000 7f16bff6b640 1 -- 172.21.15.122:0/2226511107 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b80a9b50 con 0x7f16b8140a40 2024-03-25T18:22:20.635 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.634+0000 7f16ae7fc640 1 -- 172.21.15.122:0/2226511107 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16b0002020 con 0x7f16b8140a40 2024-03-25T18:22:20.635 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.634+0000 7f16ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:20.636 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.634+0000 7f16ae7fc640 1 -- 172.21.15.122:0/2226511107 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f16b0004d80 con 0x7f16b8140a40 2024-03-25T18:22:20.636 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.634+0000 7f16ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:20.636 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.634+0000 7f16ae7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:20.636 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.634+0000 7f16ae7fc640 10 monclient: dump: 2024-03-25T18:22:20.636 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:20.636 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:20.636 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:20.636 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:20.636 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:20.636 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:20.636 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:20.636 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:20.636 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:20.636 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:20.636 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.634+0000 7f16affff640 4 set_mon_vals no callback set 2024-03-25T18:22:20.637 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.635+0000 7f16ae7fc640 1 -- 172.21.15.122:0/2226511107 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f16b001ad50 con 0x7f16b8140a40 2024-03-25T18:22:20.637 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.635+0000 7f16bdd07640 10 monclient: get_auth_request con 0x7f169c06dcb0 auth_method 0 2024-03-25T18:22:20.637 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.636+0000 7f16ae7fc640 10 client.4494.objecter ms_handle_connect 0x7f169c06dcb0 2024-03-25T18:22:20.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.644+0000 7f16ae7fc640 1 -- 172.21.15.122:0/2226511107 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f16b0054a50 con 0x7f16b8140a40 2024-03-25T18:22:20.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.644+0000 7f16ae7fc640 10 client.4494.objecter ms_dispatch 0x7f16b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:20.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.644+0000 7f16ae7fc640 3 client.4494.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:20.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.644+0000 7f16ae7fc640 3 client.4494.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:20.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.644+0000 7f16ae7fc640 20 client.4494.objecter dump_active .. 0 homeless 2024-03-25T18:22:20.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.644+0000 7f16ae7fc640 1 -- 172.21.15.122:0/2226511107 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f16b0050390 con 0x7f16b8140a40 2024-03-25T18:22:20.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.644+0000 7f16ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:20.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.644+0000 7f16ae7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:20.865 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.864+0000 7f16bff6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:20.865 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.864+0000 7f16bff6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:20.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.864+0000 7f16bff6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:22:20.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.864+0000 7f16bff6b640 1 -- 172.21.15.122:0/2226511107 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16b80627b0 con 0x7f16b8140a40 2024-03-25T18:22:20.890 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.889+0000 7f16ae7fc640 1 -- 172.21.15.122:0/2226511107 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f16b0052490 con 0x7f16b8140a40 2024-03-25T18:22:20.890 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.889+0000 7f16ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:20.890 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.889+0000 7f16ae7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:20.891 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:22:20.891 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:20.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.891+0000 7f16bff6b640 10 librados: watch_flush enter 2024-03-25T18:22:20.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.891+0000 7f16bff6b640 10 librados: watch_flush exit 2024-03-25T18:22:20.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.891+0000 7f16bff6b640 20 client.4494.objecter shutdown clearing up homeless session... 2024-03-25T18:22:20.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.891+0000 7f16bff6b640 10 client.4494.objecter successfully canceled tick 2024-03-25T18:22:20.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.891+0000 7f16bff6b640 1 -- 172.21.15.122:0/2226511107 >> v1:172.21.15.204:6800/676228005 conn(0x7f169c06dcb0 legacy=0x7f169c070170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:20.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.891+0000 7f16bff6b640 10 monclient: shutdown 2024-03-25T18:22:20.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.891+0000 7f16bff6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:20.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.891+0000 7f16bff6b640 1 -- 172.21.15.122:0/2226511107 >> v1:172.21.15.122:6789/0 conn(0x7f16b8140a40 legacy=0x7f16b81e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:20.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.891+0000 7f16bff6b640 1 -- 172.21.15.122:0/2226511107 shutdown_connections 2024-03-25T18:22:20.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.891+0000 7f16bff6b640 1 -- 172.21.15.122:0/2226511107 >> 172.21.15.122:0/2226511107 conn(0x7f16b80b5c00 msgr2=0x7f16b813f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:20.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.892+0000 7f16bff6b640 1 -- 172.21.15.122:0/2226511107 shutdown_connections 2024-03-25T18:22:20.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.892+0000 7f16bff6b640 1 -- 172.21.15.122:0/2226511107 wait complete. 2024-03-25T18:22:20.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:20.892+0000 7f16bff6b640 1 librados: shutdown 2024-03-25T18:22:20.904 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:26.499 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:06:20.241 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:26.552 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:06:20.294 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:26.584 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:06:20.326 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:26.600 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:06:20.345 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:26.603 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:06:20.346 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:26.608 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:06:20.354 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:26.609 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:06:20.352 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:26.636 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:06:20.378 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:26.643 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:06:20.388 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:27.709 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:27.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.812+0000 7f06bfa0e640 10 monclient: get_monmap_and_config 2024-03-25T18:22:27.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.812+0000 7f06bfa0e640 10 monclient: build_initial_monmap 2024-03-25T18:22:27.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.812+0000 7f06bfa0e640 10 monclient: monmap: 2024-03-25T18:22:27.813 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:22:27.813 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:27.813 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:22:27.813304+0000 2024-03-25T18:22:27.814 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:22:27.813304+0000 2024-03-25T18:22:27.814 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:27.814 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:27.814 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:22:27.814 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:22:27.814 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:22:27.814 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:27.815 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.812+0000 7f06bfa0e640 1 Processor -- start 2024-03-25T18:22:27.815 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.813+0000 7f06bfa0e640 1 -- start start 2024-03-25T18:22:27.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.813+0000 7f06bfa0e640 10 monclient: init 2024-03-25T18:22:27.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.814+0000 7f06bfa0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:27.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.814+0000 7f06bfa0e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:27.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.814+0000 7f06bfa0e640 10 monclient(hunting): picked mon.noname-c con 0x7f06b80a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:22:27.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.814+0000 7f06bfa0e640 10 monclient(hunting): picked mon.noname-b con 0x7f06b813f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:22:27.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.814+0000 7f06bfa0e640 10 monclient(hunting): picked mon.noname-a con 0x7f06b81430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:22:27.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.814+0000 7f06bfa0e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b8142520 con 0x7f06b81430c0 2024-03-25T18:22:27.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.814+0000 7f06bfa0e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b80b28a0 con 0x7f06b80a98f0 2024-03-25T18:22:27.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.814+0000 7f06bfa0e640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b80b2a30 con 0x7f06b813f710 2024-03-25T18:22:27.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.814+0000 7f06bfa0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:27.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.814+0000 7f06bfa0e640 10 monclient(hunting): authenticate will time out at 7671.080292s 2024-03-25T18:22:27.830 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.829+0000 7f06bd7aa640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f06b80a98f0 0x7f06b80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:54120/0 (socket says 172.21.15.122:54120) 2024-03-25T18:22:27.831 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.829+0000 7f06bd7aa640 1 -- 172.21.15.122:0/1698607764 learned_addr learned my addr 172.21.15.122:0/1698607764 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:22:27.853 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.851+0000 7f06affff640 1 -- 172.21.15.122:0/1698607764 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1687467159 0 0) 0x7f06b80b28a0 con 0x7f06b80a98f0 2024-03-25T18:22:27.853 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.851+0000 7f06affff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:27.853 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.851+0000 7f06affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:27.853 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.851+0000 7f06affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:27.853 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.851+0000 7f06affff640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:22:27.853 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.851+0000 7f06affff640 1 -- 172.21.15.122:0/1698607764 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06a4003520 con 0x7f06b80a98f0 2024-03-25T18:22:27.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.858+0000 7f06affff640 1 -- 172.21.15.122:0/1698607764 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3446382747 0 0) 0x7f06b80b2a30 con 0x7f06b813f710 2024-03-25T18:22:27.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.858+0000 7f06affff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:27.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.858+0000 7f06affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:27.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.858+0000 7f06affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:27.860 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.858+0000 7f06affff640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:22:27.860 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.858+0000 7f06affff640 1 -- 172.21.15.122:0/1698607764 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06b80b28a0 con 0x7f06b813f710 2024-03-25T18:22:27.862 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.861+0000 7f06affff640 1 -- 172.21.15.122:0/1698607764 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2447058280 0 0) 0x7f06b8142520 con 0x7f06b81430c0 2024-03-25T18:22:27.862 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.861+0000 7f06affff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:27.862 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.861+0000 7f06affff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:27.862 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.861+0000 7f06affff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:27.862 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.861+0000 7f06affff640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:22:27.862 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.861+0000 7f06affff640 1 -- 172.21.15.122:0/1698607764 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06b80b2a30 con 0x7f06b81430c0 2024-03-25T18:22:27.875 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.874+0000 7f06affff640 1 -- 172.21.15.122:0/1698607764 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4248742398 0 0) 0x7f06a4003520 con 0x7f06b80a98f0 2024-03-25T18:22:27.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.874+0000 7f06affff640 1 -- 172.21.15.122:0/1698607764 >> v1:172.21.15.122:6790/0 conn(0x7f06b813f710 legacy=0x7f06b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:27.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.874+0000 7f06affff640 1 -- 172.21.15.122:0/1698607764 >> v1:172.21.15.122:6789/0 conn(0x7f06b81430c0 legacy=0x7f06b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:27.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.874+0000 7f06affff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:27.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.874+0000 7f06affff640 1 monclient: found mon.noname-c 2024-03-25T18:22:27.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.874+0000 7f06affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:27.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.874+0000 7f06affff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:22:27.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.874+0000 7f06affff640 1 -- 172.21.15.122:0/1698607764 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b80b44c0 con 0x7f06b80a98f0 2024-03-25T18:22:27.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.874+0000 7f06affff640 10 monclient: _finish_auth 0 2024-03-25T18:22:27.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.874+0000 7f06affff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:27.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.874+0000 7f06affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:27.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.874+0000 7f06affff640 1 -- 172.21.15.122:0/1698607764 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f06b4001690 con 0x7f06b80a98f0 2024-03-25T18:22:27.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.874+0000 7f06affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:27.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.874+0000 7f06affff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:27.877 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.874+0000 7f06affff640 10 monclient: dump: 2024-03-25T18:22:27.877 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:27.877 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:27.877 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:27.877 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:27.877 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:27.877 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:27.877 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:27.877 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:27.877 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:27.877 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:27.877 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.874+0000 7f06bfa0e640 5 monclient: authenticate success, global_id 4495 2024-03-25T18:22:27.877 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.874+0000 7f06bfa0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:27.888 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.887+0000 7f06affff640 1 -- 172.21.15.122:0/1698607764 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06b40017f0 con 0x7f06b80a98f0 2024-03-25T18:22:27.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.887+0000 7f06affff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:27.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.887+0000 7f06affff640 1 -- 172.21.15.122:0/1698607764 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f06b4001c00 con 0x7f06b80a98f0 2024-03-25T18:22:27.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.887+0000 7f06affff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:27.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.887+0000 7f06affff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:27.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.887+0000 7f06affff640 10 monclient: dump: 2024-03-25T18:22:27.889 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:27.889 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:27.889 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:27.889 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:27.889 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:27.889 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:27.889 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:27.889 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:27.889 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:27.889 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:27.890 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.887+0000 7f06bfa0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:27.890 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.887+0000 7f06bfa0e640 4 set_mon_vals no callback set 2024-03-25T18:22:27.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.888+0000 7f06bfa0e640 10 monclient: shutdown 2024-03-25T18:22:27.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.888+0000 7f06bfa0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:27.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.888+0000 7f06bfa0e640 1 -- 172.21.15.122:0/1698607764 >> v1:172.21.15.204:6789/0 conn(0x7f06b80a98f0 legacy=0x7f06b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:27.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.888+0000 7f06bfa0e640 1 -- 172.21.15.122:0/1698607764 shutdown_connections 2024-03-25T18:22:27.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.888+0000 7f06bfa0e640 1 -- 172.21.15.122:0/1698607764 >> 172.21.15.122:0/1698607764 conn(0x7f06b80b7ea0 msgr2=0x7f06b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:27.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.888+0000 7f06bfa0e640 1 -- 172.21.15.122:0/1698607764 shutdown_connections 2024-03-25T18:22:27.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.888+0000 7f06bfa0e640 1 -- 172.21.15.122:0/1698607764 wait complete. 2024-03-25T18:22:27.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.889+0000 7f06bfa0e640 10 monclient: build_initial_monmap 2024-03-25T18:22:27.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.889+0000 7f06bfa0e640 10 monclient: monmap: 2024-03-25T18:22:27.892 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:22:27.892 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:27.892 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:22:27.892 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:22:27.892 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:27.892 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:22:27.892 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:22:27.892 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:22:27.892 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:22:27.893 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:27.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.890+0000 7f06bfa0e640 1 librados: starting msgr at 2024-03-25T18:22:27.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.890+0000 7f06bfa0e640 1 librados: starting objecter 2024-03-25T18:22:27.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.890+0000 7f06bfa0e640 1 Processor -- start 2024-03-25T18:22:27.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.890+0000 7f06bfa0e640 1 -- start start 2024-03-25T18:22:27.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.890+0000 7f06bfa0e640 1 librados: setting wanted keys 2024-03-25T18:22:27.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.890+0000 7f06bfa0e640 1 librados: calling monclient init 2024-03-25T18:22:27.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.890+0000 7f06bfa0e640 10 monclient: init 2024-03-25T18:22:27.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.891+0000 7f06bfa0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:27.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.891+0000 7f06bfa0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:27.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.891+0000 7f06bfa0e640 10 monclient(hunting): picked mon.noname-a con 0x7f06b80a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:22:27.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.891+0000 7f06bfa0e640 10 monclient(hunting): picked mon.noname-b con 0x7f06b813f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:22:27.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.891+0000 7f06bfa0e640 10 monclient(hunting): picked mon.noname-c con 0x7f06b81430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:22:27.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.891+0000 7f06bfa0e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b8138250 con 0x7f06b80a98f0 2024-03-25T18:22:27.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.891+0000 7f06bfa0e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b8137d50 con 0x7f06b813f710 2024-03-25T18:22:27.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.891+0000 7f06bfa0e640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06b8137ee0 con 0x7f06b81430c0 2024-03-25T18:22:27.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.891+0000 7f06bfa0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:27.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.891+0000 7f06bfa0e640 10 monclient(hunting): authenticate will time out at 7671.157212s 2024-03-25T18:22:27.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.893+0000 7f06bdfab640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f06b81430c0 0x7f06b81d7df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:60138/0 (socket says 172.21.15.122:60138) 2024-03-25T18:22:27.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.893+0000 7f06bdfab640 1 -- 172.21.15.122:0/3600338984 learned_addr learned my addr 172.21.15.122:0/3600338984 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:22:27.896 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.895+0000 7f06adffb640 10 client.?.objecter ms_handle_connect 0x7f06b81430c0 2024-03-25T18:22:27.896 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.895+0000 7f06adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:27.906 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.905+0000 7f06adffb640 10 client.?.objecter ms_handle_connect 0x7f06b813f710 2024-03-25T18:22:27.906 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.905+0000 7f06adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:27.907 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.906+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364743020 0 0) 0x7f06b8137ee0 con 0x7f06b81430c0 2024-03-25T18:22:27.907 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.906+0000 7f06adffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:27.907 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.906+0000 7f06adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:27.907 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.906+0000 7f06adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:27.907 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.906+0000 7f06adffb640 10 monclient(hunting): my global_id is 4469 2024-03-25T18:22:27.907 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.906+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f069c004eb0 con 0x7f06b81430c0 2024-03-25T18:22:27.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.908+0000 7f06adffb640 10 client.?.objecter ms_handle_connect 0x7f06b80a98f0 2024-03-25T18:22:27.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.908+0000 7f06adffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:27.918 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.917+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347884194 0 0) 0x7f06b8137d50 con 0x7f06b813f710 2024-03-25T18:22:27.918 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.917+0000 7f06adffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:27.918 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.917+0000 7f06adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:27.918 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.917+0000 7f06adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:27.918 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.917+0000 7f06adffb640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:22:27.918 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.917+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06b8137ee0 con 0x7f06b813f710 2024-03-25T18:22:27.927 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.926+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2258289442 0 0) 0x7f069c004eb0 con 0x7f06b81430c0 2024-03-25T18:22:27.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.926+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06b8137d50 con 0x7f06b81430c0 2024-03-25T18:22:27.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.926+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f06b0001c70 con 0x7f06b81430c0 2024-03-25T18:22:27.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.926+0000 7f06adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:27.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.926+0000 7f06adffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:27.928 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.926+0000 7f06adffb640 10 monclient(hunting): dump: 2024-03-25T18:22:27.928 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:27.928 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:27.928 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:27.928 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:27.928 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:27.928 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:27.928 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:27.928 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:27.928 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:27.928 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:27.929 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.926+0000 7f06adffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:27.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.929+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1259864505 0 0) 0x7f06b8138250 con 0x7f06b80a98f0 2024-03-25T18:22:27.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.929+0000 7f06adffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:27.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.929+0000 7f06adffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:27.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.929+0000 7f06adffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:27.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.929+0000 7f06adffb640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:22:27.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.929+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f069c004eb0 con 0x7f06b80a98f0 2024-03-25T18:22:27.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.929+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1943041844 0 0) 0x7f06b8137ee0 con 0x7f06b813f710 2024-03-25T18:22:27.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.929+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06b8138250 con 0x7f06b813f710 2024-03-25T18:22:27.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.929+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f06a8001970 con 0x7f06b813f710 2024-03-25T18:22:27.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.929+0000 7f06adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:27.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.929+0000 7f06adffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:27.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.929+0000 7f06adffb640 10 monclient(hunting): dump: 2024-03-25T18:22:27.932 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:27.932 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:27.932 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:27.932 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:27.932 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:27.932 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:27.932 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:27.932 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:27.933 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:27.933 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:27.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3458898560 0 0) 0x7f06b8137d50 con 0x7f06b81430c0 2024-03-25T18:22:27.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 >> v1:172.21.15.204:6789/0 conn(0x7f06b813f710 legacy=0x7f06b81d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:27.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 >> v1:172.21.15.122:6789/0 conn(0x7f06b80a98f0 legacy=0x7f06b8137640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:27.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06adffb640 10 monclient: _finish_hunting 0 2024-03-25T18:22:27.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06adffb640 1 monclient: found mon.c 2024-03-25T18:22:27.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:27.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06adffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:22:27.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06b80b7730 con 0x7f06b81430c0 2024-03-25T18:22:27.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06adffb640 10 monclient: _finish_auth 0 2024-03-25T18:22:27.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06adffb640 10 monclient: _check_auth_tickets 2024-03-25T18:22:27.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:27.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06bfa0e640 5 monclient: authenticate success, global_id 4469 2024-03-25T18:22:27.949 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06bfa0e640 10 monclient: _renew_subs 2024-03-25T18:22:27.949 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06bfa0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:22:27.949 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06bfa0e640 1 -- 172.21.15.122:0/3600338984 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06b80b7920 con 0x7f06b81430c0 2024-03-25T18:22:27.949 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06bfa0e640 10 client.4469.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:27.949 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06bfa0e640 10 monclient: _renew_subs 2024-03-25T18:22:27.949 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06bfa0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:22:27.949 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06bfa0e640 1 -- 172.21.15.122:0/3600338984 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06b81e33d0 con 0x7f06b81430c0 2024-03-25T18:22:27.949 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.946+0000 7f06bfa0e640 1 librados: init done 2024-03-25T18:22:27.949 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.947+0000 7f06bfa0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:27.950 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.947+0000 7f06bfa0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:27.950 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.947+0000 7f06bfa0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:22:27.950 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:27.947+0000 7f06bfa0e640 1 -- 172.21.15.122:0/3600338984 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06b8139770 con 0x7f06b81430c0 2024-03-25T18:22:28.252 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.251+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06b0003970 con 0x7f06b81430c0 2024-03-25T18:22:28.252 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.251+0000 7f06adffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:28.252 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.251+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f06b0003be0 con 0x7f06b81430c0 2024-03-25T18:22:28.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.251+0000 7f06adffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:28.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.251+0000 7f06adffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:28.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.251+0000 7f06adffb640 10 monclient: dump: 2024-03-25T18:22:28.253 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:28.253 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:28.253 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:28.253 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:28.253 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:28.253 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:28.253 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:28.253 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:28.254 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:28.254 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:28.254 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.251+0000 7f06af7fe640 4 set_mon_vals no callback set 2024-03-25T18:22:28.254 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.252+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f06b0003e00 con 0x7f06b81430c0 2024-03-25T18:22:28.254 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.253+0000 7f06bd7aa640 10 monclient: get_auth_request con 0x7f069c06dfb0 auth_method 0 2024-03-25T18:22:28.255 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.253+0000 7f06adffb640 10 client.4469.objecter ms_handle_connect 0x7f069c06dfb0 2024-03-25T18:22:28.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.262+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f06b0054930 con 0x7f06b81430c0 2024-03-25T18:22:28.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.262+0000 7f06adffb640 10 client.4469.objecter ms_dispatch 0x7f06b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:28.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.262+0000 7f06adffb640 3 client.4469.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:28.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.262+0000 7f06adffb640 3 client.4469.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:28.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.262+0000 7f06adffb640 20 client.4469.objecter dump_active .. 0 homeless 2024-03-25T18:22:28.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.262+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f06b0050440 con 0x7f06b81430c0 2024-03-25T18:22:28.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.262+0000 7f06adffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:28.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.262+0000 7f06adffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:28.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.514+0000 7f06bfa0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:28.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.514+0000 7f06bfa0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:28.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.514+0000 7f06bfa0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:22:28.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.514+0000 7f06bfa0e640 1 -- 172.21.15.122:0/3600338984 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06b80627b0 con 0x7f06b81430c0 2024-03-25T18:22:28.560 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.559+0000 7f06adffb640 1 -- 172.21.15.122:0/3600338984 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f06b0004800 con 0x7f06b81430c0 2024-03-25T18:22:28.560 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.559+0000 7f06adffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:28.560 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.559+0000 7f06adffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:28.560 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:22:28.560 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:28.563 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.561+0000 7f06bfa0e640 10 librados: watch_flush enter 2024-03-25T18:22:28.563 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.561+0000 7f06bfa0e640 10 librados: watch_flush exit 2024-03-25T18:22:28.563 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.561+0000 7f06bfa0e640 20 client.4469.objecter shutdown clearing up homeless session... 2024-03-25T18:22:28.563 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.561+0000 7f06bfa0e640 10 client.4469.objecter successfully canceled tick 2024-03-25T18:22:28.563 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.561+0000 7f06bfa0e640 1 -- 172.21.15.122:0/3600338984 >> v1:172.21.15.204:6800/676228005 conn(0x7f069c06dfb0 legacy=0x7f069c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:28.563 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.561+0000 7f06bfa0e640 10 monclient: shutdown 2024-03-25T18:22:28.563 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.561+0000 7f06bfa0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:28.563 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.561+0000 7f06bfa0e640 1 -- 172.21.15.122:0/3600338984 >> v1:172.21.15.122:6790/0 conn(0x7f06b81430c0 legacy=0x7f06b81d7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:28.563 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.561+0000 7f06bfa0e640 1 -- 172.21.15.122:0/3600338984 shutdown_connections 2024-03-25T18:22:28.563 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.561+0000 7f06bfa0e640 1 -- 172.21.15.122:0/3600338984 >> 172.21.15.122:0/3600338984 conn(0x7f06b80b7ea0 msgr2=0x7f06b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:28.563 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.561+0000 7f06bfa0e640 1 -- 172.21.15.122:0/3600338984 shutdown_connections 2024-03-25T18:22:28.563 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.561+0000 7f06bfa0e640 1 -- 172.21.15.122:0/3600338984 wait complete. 2024-03-25T18:22:28.563 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:28.561+0000 7f06bfa0e640 1 librados: shutdown 2024-03-25T18:22:28.576 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:35.384 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:35.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.495+0000 7f610ac50640 10 monclient: get_monmap_and_config 2024-03-25T18:22:35.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.495+0000 7f610ac50640 10 monclient: build_initial_monmap 2024-03-25T18:22:35.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.495+0000 7f610ac50640 10 monclient: monmap: 2024-03-25T18:22:35.496 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:22:35.496 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:35.496 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:22:35.496304+0000 2024-03-25T18:22:35.496 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:22:35.496304+0000 2024-03-25T18:22:35.497 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:35.497 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:35.497 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:22:35.497 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:22:35.497 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:22:35.497 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:35.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.496+0000 7f610ac50640 1 Processor -- start 2024-03-25T18:22:35.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.496+0000 7f610ac50640 1 -- start start 2024-03-25T18:22:35.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.496+0000 7f610ac50640 10 monclient: init 2024-03-25T18:22:35.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.497+0000 7f610ac50640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:35.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.497+0000 7f610ac50640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:35.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.497+0000 7f610ac50640 10 monclient(hunting): picked mon.noname-a con 0x7f61040a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:22:35.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.497+0000 7f610ac50640 10 monclient(hunting): picked mon.noname-c con 0x7f610413f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:22:35.499 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.497+0000 7f610ac50640 10 monclient(hunting): picked mon.noname-b con 0x7f61041430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:22:35.499 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.497+0000 7f610ac50640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6104142520 con 0x7f61040a98f0 2024-03-25T18:22:35.499 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.497+0000 7f610ac50640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61040b28a0 con 0x7f610413f710 2024-03-25T18:22:35.499 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.497+0000 7f610ac50640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61040b2a30 con 0x7f61041430c0 2024-03-25T18:22:35.499 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.497+0000 7f610ac50640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:35.499 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.497+0000 7f610ac50640 10 monclient(hunting): authenticate will time out at 7678.763270s 2024-03-25T18:22:35.503 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.502+0000 7f61091ed640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f61041430c0 0x7f610414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:59116/0 (socket says 172.21.15.122:59116) 2024-03-25T18:22:35.503 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.502+0000 7f61091ed640 1 -- 172.21.15.122:0/4165544078 learned_addr learned my addr 172.21.15.122:0/4165544078 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.519+0000 7f61037fe640 1 -- 172.21.15.122:0/4165544078 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576323751 0 0) 0x7f61040b28a0 con 0x7f610413f710 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.519+0000 7f61037fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.519+0000 7f61037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.519+0000 7f61037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.519+0000 7f61037fe640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:22:35.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.519+0000 7f61037fe640 1 -- 172.21.15.122:0/4165544078 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60f4003520 con 0x7f610413f710 2024-03-25T18:22:35.524 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.522+0000 7f61037fe640 1 -- 172.21.15.122:0/4165544078 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3329958457 0 0) 0x7f6104142520 con 0x7f61040a98f0 2024-03-25T18:22:35.524 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.523+0000 7f61037fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:35.524 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.523+0000 7f61037fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:35.524 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.523+0000 7f61037fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:35.524 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.523+0000 7f61037fe640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:22:35.524 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.523+0000 7f61037fe640 1 -- 172.21.15.122:0/4165544078 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61040b28a0 con 0x7f61040a98f0 2024-03-25T18:22:35.530 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f61037fe640 1 -- 172.21.15.122:0/4165544078 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4152844902 0 0) 0x7f60f4003520 con 0x7f610413f710 2024-03-25T18:22:35.530 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f61037fe640 1 -- 172.21.15.122:0/4165544078 >> v1:172.21.15.122:6790/0 conn(0x7f61041430c0 legacy=0x7f610414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:35.530 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f61037fe640 1 -- 172.21.15.122:0/4165544078 >> v1:172.21.15.122:6789/0 conn(0x7f61040a98f0 legacy=0x7f61040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:35.530 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f61037fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:35.530 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f61037fe640 1 monclient: found mon.noname-c 2024-03-25T18:22:35.530 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f61037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:35.530 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f61037fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:22:35.530 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f61037fe640 1 -- 172.21.15.122:0/4165544078 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61040b44c0 con 0x7f610413f710 2024-03-25T18:22:35.530 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f61037fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:35.530 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f61037fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:35.530 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f61037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:35.530 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f61037fe640 1 -- 172.21.15.122:0/4165544078 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f60f0001690 con 0x7f610413f710 2024-03-25T18:22:35.531 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f610ac50640 5 monclient: authenticate success, global_id 4501 2024-03-25T18:22:35.531 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f610ac50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:35.531 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f61037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:35.531 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f61037fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:35.531 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f61037fe640 10 monclient: dump: 2024-03-25T18:22:35.531 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:35.531 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:35.531 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:35.531 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:35.531 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:35.531 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:35.531 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:35.531 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:35.531 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:35.531 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:35.531 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.528+0000 7f610ac50640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.540+0000 7f61037fe640 1 -- 172.21.15.122:0/4165544078 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60f00017f0 con 0x7f610413f710 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.540+0000 7f61037fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.540+0000 7f61037fe640 1 -- 172.21.15.122:0/4165544078 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f60f0001c40 con 0x7f610413f710 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.540+0000 7f61037fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.540+0000 7f61037fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.540+0000 7f61037fe640 10 monclient: dump: 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:35.541 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.540+0000 7f610ac50640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:35.542 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.540+0000 7f610ac50640 4 set_mon_vals no callback set 2024-03-25T18:22:35.543 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.541+0000 7f610ac50640 10 monclient: shutdown 2024-03-25T18:22:35.543 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.541+0000 7f610ac50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:35.543 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.541+0000 7f610ac50640 1 -- 172.21.15.122:0/4165544078 >> v1:172.21.15.204:6789/0 conn(0x7f610413f710 legacy=0x7f61041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:35.543 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.541+0000 7f610ac50640 1 -- 172.21.15.122:0/4165544078 shutdown_connections 2024-03-25T18:22:35.543 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.541+0000 7f610ac50640 1 -- 172.21.15.122:0/4165544078 >> 172.21.15.122:0/4165544078 conn(0x7f61040b7ea0 msgr2=0x7f61040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:35.543 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.541+0000 7f610ac50640 1 -- 172.21.15.122:0/4165544078 shutdown_connections 2024-03-25T18:22:35.543 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.541+0000 7f610ac50640 1 -- 172.21.15.122:0/4165544078 wait complete. 2024-03-25T18:22:35.543 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.542+0000 7f610ac50640 10 monclient: build_initial_monmap 2024-03-25T18:22:35.544 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.542+0000 7f610ac50640 10 monclient: monmap: 2024-03-25T18:22:35.544 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:22:35.544 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:35.544 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:22:35.544 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:22:35.544 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:35.544 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:22:35.544 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:22:35.544 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:22:35.544 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:22:35.544 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:35.545 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.542+0000 7f610ac50640 1 librados: starting msgr at 2024-03-25T18:22:35.545 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.542+0000 7f610ac50640 1 librados: starting objecter 2024-03-25T18:22:35.545 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.542+0000 7f610ac50640 1 Processor -- start 2024-03-25T18:22:35.545 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.543+0000 7f610ac50640 1 -- start start 2024-03-25T18:22:35.545 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.543+0000 7f610ac50640 1 librados: setting wanted keys 2024-03-25T18:22:35.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.543+0000 7f610ac50640 1 librados: calling monclient init 2024-03-25T18:22:35.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.543+0000 7f610ac50640 10 monclient: init 2024-03-25T18:22:35.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.543+0000 7f610ac50640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:35.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.543+0000 7f610ac50640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:35.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.543+0000 7f610ac50640 10 monclient(hunting): picked mon.noname-b con 0x7f61040a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:22:35.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.543+0000 7f610ac50640 10 monclient(hunting): picked mon.noname-c con 0x7f610413f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:22:35.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.543+0000 7f610ac50640 10 monclient(hunting): picked mon.noname-a con 0x7f61041430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:22:35.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.543+0000 7f610ac50640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6104138bf0 con 0x7f61041430c0 2024-03-25T18:22:35.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.543+0000 7f610ac50640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61041d7750 con 0x7f61040a98f0 2024-03-25T18:22:35.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.543+0000 7f610ac50640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61040abb00 con 0x7f610413f710 2024-03-25T18:22:35.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.543+0000 7f610ac50640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:35.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.543+0000 7f610ac50640 10 monclient(hunting): authenticate will time out at 7678.809942s 2024-03-25T18:22:35.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.555+0000 7f61089ec640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f61040a98f0 0x7f61041384e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:39740/0 (socket says 172.21.15.122:39740) 2024-03-25T18:22:35.557 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.555+0000 7f61089ec640 1 -- 172.21.15.122:0/1853587791 learned_addr learned my addr 172.21.15.122:0/1853587791 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:22:35.559 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.558+0000 7f61017fa640 10 client.?.objecter ms_handle_connect 0x7f61040a98f0 2024-03-25T18:22:35.559 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.558+0000 7f61017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:35.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.567+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762587843 0 0) 0x7f61041d7750 con 0x7f61040a98f0 2024-03-25T18:22:35.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.567+0000 7f61017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:35.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.567+0000 7f61017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:35.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.567+0000 7f61017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:35.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.567+0000 7f61017fa640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:22:35.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.567+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e8004eb0 con 0x7f61040a98f0 2024-03-25T18:22:35.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.568+0000 7f61017fa640 10 client.?.objecter ms_handle_connect 0x7f610413f710 2024-03-25T18:22:35.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.568+0000 7f61017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:35.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.573+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1111154548 0 0) 0x7f60e8004eb0 con 0x7f61040a98f0 2024-03-25T18:22:35.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.573+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61041d7750 con 0x7f61040a98f0 2024-03-25T18:22:35.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.573+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f60f8001a10 con 0x7f61040a98f0 2024-03-25T18:22:35.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.573+0000 7f61017fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:35.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.573+0000 7f61017fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:35.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.573+0000 7f61017fa640 10 monclient(hunting): dump: 2024-03-25T18:22:35.575 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:35.575 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:35.575 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:35.575 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:35.575 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:35.576 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:35.576 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:35.576 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:35.576 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:35.576 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:35.576 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.573+0000 7f61017fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:35.576 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.574+0000 7f61017fa640 10 client.?.objecter ms_handle_connect 0x7f61041430c0 2024-03-25T18:22:35.576 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.574+0000 7f61017fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:35.582 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.580+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357353722 0 0) 0x7f61040abb00 con 0x7f610413f710 2024-03-25T18:22:35.582 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.580+0000 7f61017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:35.582 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.580+0000 7f61017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:35.582 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.580+0000 7f61017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:35.582 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.580+0000 7f61017fa640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:22:35.582 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.580+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e8004eb0 con 0x7f610413f710 2024-03-25T18:22:35.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3644854996 0 0) 0x7f6104138bf0 con 0x7f61041430c0 2024-03-25T18:22:35.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f61017fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:35.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f61017fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:35.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f61017fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:35.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f61017fa640 10 monclient(hunting): my global_id is 4506 2024-03-25T18:22:35.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61040abb00 con 0x7f61041430c0 2024-03-25T18:22:35.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 501938880 0 0) 0x7f61041d7750 con 0x7f61040a98f0 2024-03-25T18:22:35.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 >> v1:172.21.15.122:6790/0 conn(0x7f610413f710 legacy=0x7f61041d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:35.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 >> v1:172.21.15.122:6789/0 conn(0x7f61041430c0 legacy=0x7f6104133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:35.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f61017fa640 10 monclient: _finish_hunting 0 2024-03-25T18:22:35.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f61017fa640 1 monclient: found mon.b 2024-03-25T18:22:35.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f61017fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:35.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f61017fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:22:35.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61041d41e0 con 0x7f61040a98f0 2024-03-25T18:22:35.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f61017fa640 10 monclient: _finish_auth 0 2024-03-25T18:22:35.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f61017fa640 10 monclient: _check_auth_tickets 2024-03-25T18:22:35.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f61017fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:35.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f610ac50640 5 monclient: authenticate success, global_id 4504 2024-03-25T18:22:35.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f610ac50640 10 monclient: _renew_subs 2024-03-25T18:22:35.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f610ac50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:22:35.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f610ac50640 1 -- 172.21.15.122:0/1853587791 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6104137a90 con 0x7f61040a98f0 2024-03-25T18:22:35.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f610ac50640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:35.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f610ac50640 10 monclient: _renew_subs 2024-03-25T18:22:35.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f610ac50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:22:35.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f610ac50640 1 -- 172.21.15.122:0/1853587791 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f61041d82f0 con 0x7f61040a98f0 2024-03-25T18:22:35.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.584+0000 7f610ac50640 1 librados: init done 2024-03-25T18:22:35.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.585+0000 7f610ac50640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:35.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.585+0000 7f610ac50640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:35.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.585+0000 7f610ac50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:22:35.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.585+0000 7f610ac50640 1 -- 172.21.15.122:0/1853587791 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61040627b0 con 0x7f61040a98f0 2024-03-25T18:22:35.853 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.852+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60f8001cd0 con 0x7f61040a98f0 2024-03-25T18:22:35.853 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.852+0000 7f61017fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:35.853 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.852+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f60f8003a60 con 0x7f61040a98f0 2024-03-25T18:22:35.853 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.852+0000 7f61017fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:35.853 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.852+0000 7f61017fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:35.853 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.852+0000 7f61017fa640 10 monclient: dump: 2024-03-25T18:22:35.854 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:35.854 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:35.854 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:35.854 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:35.854 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:35.854 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:35.854 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:35.854 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:35.854 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:35.854 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:35.854 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.852+0000 7f6102ffd640 4 set_mon_vals no callback set 2024-03-25T18:22:35.855 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.853+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f60f8003c80 con 0x7f61040a98f0 2024-03-25T18:22:35.855 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.854+0000 7f6103fff640 10 monclient: get_auth_request con 0x7f60e806ddc0 auth_method 0 2024-03-25T18:22:35.855 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.854+0000 7f61017fa640 10 client.4504.objecter ms_handle_connect 0x7f60e806ddc0 2024-03-25T18:22:35.857 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.856+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f60f8054930 con 0x7f61040a98f0 2024-03-25T18:22:35.857 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.856+0000 7f61017fa640 10 client.4504.objecter ms_dispatch 0x7f6104000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:35.857 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.856+0000 7f61017fa640 3 client.4504.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:35.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.856+0000 7f61017fa640 3 client.4504.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:35.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.856+0000 7f61017fa640 20 client.4504.objecter dump_active .. 0 homeless 2024-03-25T18:22:35.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.858+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f60f8050270 con 0x7f61040a98f0 2024-03-25T18:22:35.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.858+0000 7f61017fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:35.860 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:35.858+0000 7f61017fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:36.111 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.110+0000 7f610ac50640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:36.111 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.110+0000 7f610ac50640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:36.111 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.110+0000 7f610ac50640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:22:36.111 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.110+0000 7f610ac50640 1 -- 172.21.15.122:0/1853587791 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f610413dfd0 con 0x7f61040a98f0 2024-03-25T18:22:36.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.138+0000 7f61017fa640 1 -- 172.21.15.122:0/1853587791 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f60f8052370 con 0x7f61040a98f0 2024-03-25T18:22:36.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.138+0000 7f61017fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:36.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.138+0000 7f61017fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:36.139 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:22:36.139 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:36.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.140+0000 7f610ac50640 10 librados: watch_flush enter 2024-03-25T18:22:36.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.140+0000 7f610ac50640 10 librados: watch_flush exit 2024-03-25T18:22:36.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.140+0000 7f610ac50640 20 client.4504.objecter shutdown clearing up homeless session... 2024-03-25T18:22:36.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.140+0000 7f610ac50640 10 client.4504.objecter successfully canceled tick 2024-03-25T18:22:36.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.140+0000 7f610ac50640 1 -- 172.21.15.122:0/1853587791 >> v1:172.21.15.204:6800/676228005 conn(0x7f60e806ddc0 legacy=0x7f60e8070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:36.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.140+0000 7f610ac50640 10 monclient: shutdown 2024-03-25T18:22:36.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.140+0000 7f610ac50640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:36.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.140+0000 7f610ac50640 1 -- 172.21.15.122:0/1853587791 >> v1:172.21.15.204:6789/0 conn(0x7f61040a98f0 legacy=0x7f61041384e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:36.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.140+0000 7f610ac50640 1 -- 172.21.15.122:0/1853587791 shutdown_connections 2024-03-25T18:22:36.143 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.140+0000 7f610ac50640 1 -- 172.21.15.122:0/1853587791 >> 172.21.15.122:0/1853587791 conn(0x7f61040b7ea0 msgr2=0x7f6104074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:36.143 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.140+0000 7f610ac50640 1 -- 172.21.15.122:0/1853587791 shutdown_connections 2024-03-25T18:22:36.143 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.140+0000 7f610ac50640 1 -- 172.21.15.122:0/1853587791 wait complete. 2024-03-25T18:22:36.143 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:36.140+0000 7f610ac50640 1 librados: shutdown 2024-03-25T18:22:36.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:42.960 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:43.075 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.075+0000 7f23db065640 10 monclient: get_monmap_and_config 2024-03-25T18:22:43.075 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.075+0000 7f23db065640 10 monclient: build_initial_monmap 2024-03-25T18:22:43.075 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.075+0000 7f23db065640 10 monclient: monmap: 2024-03-25T18:22:43.075 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:22:43.075 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:43.075 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:22:43.075237+0000 2024-03-25T18:22:43.075 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:22:43.075237+0000 2024-03-25T18:22:43.076 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:43.076 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:43.076 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:22:43.076 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:22:43.076 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:22:43.076 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:43.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.075+0000 7f23db065640 1 Processor -- start 2024-03-25T18:22:43.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.075+0000 7f23db065640 1 -- start start 2024-03-25T18:22:43.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.075+0000 7f23db065640 10 monclient: init 2024-03-25T18:22:43.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.076+0000 7f23db065640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:43.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.076+0000 7f23db065640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:43.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.076+0000 7f23db065640 10 monclient(hunting): picked mon.noname-c con 0x7f23d40a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:22:43.078 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.076+0000 7f23db065640 10 monclient(hunting): picked mon.noname-a con 0x7f23d413f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:22:43.078 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.076+0000 7f23db065640 10 monclient(hunting): picked mon.noname-b con 0x7f23d41430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:22:43.078 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.076+0000 7f23db065640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d4142520 con 0x7f23d413f710 2024-03-25T18:22:43.078 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.076+0000 7f23db065640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d40b28a0 con 0x7f23d40a98f0 2024-03-25T18:22:43.078 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.076+0000 7f23db065640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d40b2a30 con 0x7f23d41430c0 2024-03-25T18:22:43.078 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.076+0000 7f23db065640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:43.078 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.076+0000 7f23db065640 10 monclient(hunting): authenticate will time out at 7686.341597s 2024-03-25T18:22:43.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.079+0000 7f23d3fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f23d413f710 0x7f23d41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:39968/0 (socket says 172.21.15.122:39968) 2024-03-25T18:22:43.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.080+0000 7f23d3fff640 1 -- 172.21.15.122:0/3017576222 learned_addr learned my addr 172.21.15.122:0/3017576222 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:22:43.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.097+0000 7f23d37fe640 1 -- 172.21.15.122:0/3017576222 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631137789 0 0) 0x7f23d40b2a30 con 0x7f23d41430c0 2024-03-25T18:22:43.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.097+0000 7f23d37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:43.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.097+0000 7f23d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:43.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.097+0000 7f23d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:43.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.097+0000 7f23d37fe640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:22:43.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.097+0000 7f23d37fe640 1 -- 172.21.15.122:0/3017576222 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23c4003520 con 0x7f23d41430c0 2024-03-25T18:22:43.105 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.105+0000 7f23d37fe640 1 -- 172.21.15.122:0/3017576222 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171316838 0 0) 0x7f23d40b28a0 con 0x7f23d40a98f0 2024-03-25T18:22:43.105 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.105+0000 7f23d37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:43.105 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.105+0000 7f23d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:43.105 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.105+0000 7f23d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:43.105 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.105+0000 7f23d37fe640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:22:43.106 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.105+0000 7f23d37fe640 1 -- 172.21.15.122:0/3017576222 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23d40b2a30 con 0x7f23d40a98f0 2024-03-25T18:22:43.110 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.110+0000 7f23d37fe640 1 -- 172.21.15.122:0/3017576222 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972751820 0 0) 0x7f23d4142520 con 0x7f23d413f710 2024-03-25T18:22:43.110 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.110+0000 7f23d37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:43.110 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.110+0000 7f23d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:43.110 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.110+0000 7f23d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:43.110 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.110+0000 7f23d37fe640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:22:43.110 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.110+0000 7f23d37fe640 1 -- 172.21.15.122:0/3017576222 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23d40b28a0 con 0x7f23d413f710 2024-03-25T18:22:43.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.119+0000 7f23d37fe640 1 -- 172.21.15.122:0/3017576222 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 108256294 0 0) 0x7f23c4003520 con 0x7f23d41430c0 2024-03-25T18:22:43.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.119+0000 7f23d37fe640 1 -- 172.21.15.122:0/3017576222 >> v1:172.21.15.204:6789/0 conn(0x7f23d40a98f0 legacy=0x7f23d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:43.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.119+0000 7f23d37fe640 1 -- 172.21.15.122:0/3017576222 >> v1:172.21.15.122:6789/0 conn(0x7f23d413f710 legacy=0x7f23d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:43.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.119+0000 7f23d37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:43.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.119+0000 7f23d37fe640 1 monclient: found mon.noname-b 2024-03-25T18:22:43.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.119+0000 7f23d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:43.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.119+0000 7f23d37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:22:43.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.119+0000 7f23d37fe640 1 -- 172.21.15.122:0/3017576222 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d40b44c0 con 0x7f23d41430c0 2024-03-25T18:22:43.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.119+0000 7f23d37fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:43.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.119+0000 7f23d37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:43.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.119+0000 7f23d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:43.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.119+0000 7f23d37fe640 1 -- 172.21.15.122:0/3017576222 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f23cc001a10 con 0x7f23d41430c0 2024-03-25T18:22:43.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.119+0000 7f23d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:43.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.119+0000 7f23d37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:43.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.119+0000 7f23d37fe640 10 monclient: dump: 2024-03-25T18:22:43.121 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:43.121 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:43.121 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:43.121 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:43.121 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:43.121 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:43.121 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:43.121 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:43.121 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:43.121 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:43.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.119+0000 7f23db065640 5 monclient: authenticate success, global_id 4478 2024-03-25T18:22:43.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.119+0000 7f23db065640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:43.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.131+0000 7f23d37fe640 1 -- 172.21.15.122:0/3017576222 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23cc001b70 con 0x7f23d41430c0 2024-03-25T18:22:43.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.131+0000 7f23d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:43.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.131+0000 7f23d37fe640 1 -- 172.21.15.122:0/3017576222 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f23cc001f80 con 0x7f23d41430c0 2024-03-25T18:22:43.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.131+0000 7f23d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:43.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.131+0000 7f23d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:43.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.131+0000 7f23d37fe640 10 monclient: dump: 2024-03-25T18:22:43.131 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:43.131 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:43.132 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:43.132 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:43.132 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:43.132 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:43.132 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:43.132 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:43.132 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:43.132 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:43.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.131+0000 7f23db065640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:43.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.131+0000 7f23db065640 4 set_mon_vals no callback set 2024-03-25T18:22:43.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.131+0000 7f23db065640 10 monclient: shutdown 2024-03-25T18:22:43.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.131+0000 7f23db065640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:43.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.131+0000 7f23db065640 1 -- 172.21.15.122:0/3017576222 >> v1:172.21.15.122:6790/0 conn(0x7f23d41430c0 legacy=0x7f23d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:43.134 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.131+0000 7f23db065640 1 -- 172.21.15.122:0/3017576222 shutdown_connections 2024-03-25T18:22:43.134 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.131+0000 7f23db065640 1 -- 172.21.15.122:0/3017576222 >> 172.21.15.122:0/3017576222 conn(0x7f23d40b7ea0 msgr2=0x7f23d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:43.134 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.132+0000 7f23db065640 1 -- 172.21.15.122:0/3017576222 shutdown_connections 2024-03-25T18:22:43.134 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.132+0000 7f23db065640 1 -- 172.21.15.122:0/3017576222 wait complete. 2024-03-25T18:22:43.134 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.132+0000 7f23db065640 10 monclient: build_initial_monmap 2024-03-25T18:22:43.134 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.132+0000 7f23db065640 10 monclient: monmap: 2024-03-25T18:22:43.134 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:22:43.134 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:43.134 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:22:43.134 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:22:43.134 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:43.134 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:22:43.134 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:22:43.134 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:22:43.135 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:22:43.135 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:43.135 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.133+0000 7f23db065640 1 librados: starting msgr at 2024-03-25T18:22:43.135 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.133+0000 7f23db065640 1 librados: starting objecter 2024-03-25T18:22:43.135 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.133+0000 7f23db065640 1 Processor -- start 2024-03-25T18:22:43.135 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.133+0000 7f23db065640 1 -- start start 2024-03-25T18:22:43.135 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.133+0000 7f23db065640 1 librados: setting wanted keys 2024-03-25T18:22:43.135 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.133+0000 7f23db065640 1 librados: calling monclient init 2024-03-25T18:22:43.135 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.133+0000 7f23db065640 10 monclient: init 2024-03-25T18:22:43.136 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.134+0000 7f23db065640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:43.136 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.134+0000 7f23db065640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:43.136 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.134+0000 7f23db065640 10 monclient(hunting): picked mon.noname-b con 0x7f23d40a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:22:43.136 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.134+0000 7f23db065640 10 monclient(hunting): picked mon.noname-c con 0x7f23d413f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:22:43.136 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.134+0000 7f23db065640 10 monclient(hunting): picked mon.noname-a con 0x7f23d41430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:22:43.136 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.134+0000 7f23db065640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d413c9b0 con 0x7f23d41430c0 2024-03-25T18:22:43.136 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.134+0000 7f23db065640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d41397c0 con 0x7f23d40a98f0 2024-03-25T18:22:43.136 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.134+0000 7f23db065640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23d4139950 con 0x7f23d413f710 2024-03-25T18:22:43.136 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.134+0000 7f23db065640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:43.136 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.134+0000 7f23db065640 10 monclient(hunting): authenticate will time out at 7686.399430s 2024-03-25T18:22:43.148 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.148+0000 7f23d3fff640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f23d413f710 0x7f23d41dae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:52562/0 (socket says 172.21.15.122:52562) 2024-03-25T18:22:43.148 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.148+0000 7f23d3fff640 1 -- 172.21.15.122:0/2444483398 learned_addr learned my addr 172.21.15.122:0/2444483398 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:22:43.154 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.153+0000 7f23d17fa640 10 client.?.objecter ms_handle_connect 0x7f23d413f710 2024-03-25T18:22:43.154 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.153+0000 7f23d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:43.154 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.154+0000 7f23d17fa640 10 client.?.objecter ms_handle_connect 0x7f23d40a98f0 2024-03-25T18:22:43.154 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.154+0000 7f23d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:43.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.164+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 900784062 0 0) 0x7f23d41397c0 con 0x7f23d40a98f0 2024-03-25T18:22:43.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.164+0000 7f23d17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:43.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.164+0000 7f23d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:43.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.164+0000 7f23d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:43.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.164+0000 7f23d17fa640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:22:43.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.164+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b8004eb0 con 0x7f23d40a98f0 2024-03-25T18:22:43.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.165+0000 7f23d17fa640 10 client.?.objecter ms_handle_connect 0x7f23d41430c0 2024-03-25T18:22:43.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.165+0000 7f23d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:43.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.167+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1604600557 0 0) 0x7f23d4139950 con 0x7f23d413f710 2024-03-25T18:22:43.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.167+0000 7f23d17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:43.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.167+0000 7f23d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:43.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.167+0000 7f23d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:43.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.167+0000 7f23d17fa640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:22:43.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.167+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23d41397c0 con 0x7f23d413f710 2024-03-25T18:22:43.177 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.176+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 71839813 0 0) 0x7f23b8004eb0 con 0x7f23d40a98f0 2024-03-25T18:22:43.177 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.177+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23d4139950 con 0x7f23d40a98f0 2024-03-25T18:22:43.177 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.177+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f23c8001970 con 0x7f23d40a98f0 2024-03-25T18:22:43.177 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.177+0000 7f23d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:43.177 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.177+0000 7f23d17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:43.177 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.177+0000 7f23d17fa640 10 monclient(hunting): dump: 2024-03-25T18:22:43.177 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:43.177 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:43.177 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:43.178 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:43.178 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:43.178 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:43.178 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:43.178 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:43.178 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:43.178 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:43.178 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.177+0000 7f23d17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:43.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.181+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3038047567 0 0) 0x7f23d41397c0 con 0x7f23d413f710 2024-03-25T18:22:43.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.181+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23b8004eb0 con 0x7f23d413f710 2024-03-25T18:22:43.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.181+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f23c00018f0 con 0x7f23d413f710 2024-03-25T18:22:43.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.181+0000 7f23d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:43.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.181+0000 7f23d17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:43.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.181+0000 7f23d17fa640 10 monclient(hunting): dump: 2024-03-25T18:22:43.182 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:43.182 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:43.182 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:43.182 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:43.183 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:43.183 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:43.183 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:43.183 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:43.183 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:43.183 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:43.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.183+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1884695106 0 0) 0x7f23d413c9b0 con 0x7f23d41430c0 2024-03-25T18:22:43.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.183+0000 7f23d17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:43.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.183+0000 7f23d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:43.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.183+0000 7f23d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:43.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.183+0000 7f23d17fa640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:22:43.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.183+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23d41397c0 con 0x7f23d41430c0 2024-03-25T18:22:43.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.187+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 446085742 0 0) 0x7f23d4139950 con 0x7f23d40a98f0 2024-03-25T18:22:43.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.187+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 >> v1:172.21.15.122:6790/0 conn(0x7f23d413f710 legacy=0x7f23d41dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:43.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.187+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 >> v1:172.21.15.122:6789/0 conn(0x7f23d41430c0 legacy=0x7f23d41e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:43.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.187+0000 7f23d17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:22:43.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.187+0000 7f23d17fa640 1 monclient: found mon.b 2024-03-25T18:22:43.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.187+0000 7f23d17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:43.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.187+0000 7f23d17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:22:43.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.187+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23d4131680 con 0x7f23d40a98f0 2024-03-25T18:22:43.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.187+0000 7f23d17fa640 10 monclient: _finish_auth 0 2024-03-25T18:22:43.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.187+0000 7f23d17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:22:43.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.187+0000 7f23d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:43.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.188+0000 7f23db065640 5 monclient: authenticate success, global_id 4510 2024-03-25T18:22:43.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.188+0000 7f23db065640 10 monclient: _renew_subs 2024-03-25T18:22:43.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.188+0000 7f23db065640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:22:43.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.188+0000 7f23db065640 1 -- 172.21.15.122:0/2444483398 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d4131950 con 0x7f23d40a98f0 2024-03-25T18:22:43.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.188+0000 7f23db065640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:43.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.188+0000 7f23db065640 10 monclient: _renew_subs 2024-03-25T18:22:43.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.188+0000 7f23db065640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:22:43.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.188+0000 7f23db065640 1 -- 172.21.15.122:0/2444483398 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d4135a30 con 0x7f23d40a98f0 2024-03-25T18:22:43.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.188+0000 7f23db065640 1 librados: init done 2024-03-25T18:22:43.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.189+0000 7f23db065640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:43.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.189+0000 7f23db065640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:43.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.189+0000 7f23db065640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:22:43.192 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.189+0000 7f23db065640 1 -- 172.21.15.122:0/2444483398 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d413c810 con 0x7f23d40a98f0 2024-03-25T18:22:43.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.490+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23c80035f0 con 0x7f23d40a98f0 2024-03-25T18:22:43.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.490+0000 7f23d17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:43.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.490+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f23c8003760 con 0x7f23d40a98f0 2024-03-25T18:22:43.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.490+0000 7f23d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:43.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.491+0000 7f23d17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:43.491 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.491+0000 7f23d17fa640 10 monclient: dump: 2024-03-25T18:22:43.491 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:43.491 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:43.492 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:43.492 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:43.492 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:43.492 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:43.492 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:43.492 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:43.492 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:43.492 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:43.492 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.490+0000 7f23d2ffd640 4 set_mon_vals no callback set 2024-03-25T18:22:43.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.491+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f23c8019730 con 0x7f23d40a98f0 2024-03-25T18:22:43.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.492+0000 7f23d3fff640 10 monclient: get_auth_request con 0x7f23b806deb0 auth_method 0 2024-03-25T18:22:43.493 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.492+0000 7f23d17fa640 10 client.4510.objecter ms_handle_connect 0x7f23b806deb0 2024-03-25T18:22:43.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.497+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f23c80533d0 con 0x7f23d40a98f0 2024-03-25T18:22:43.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.497+0000 7f23d17fa640 10 client.4510.objecter ms_dispatch 0x7f23d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:43.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.497+0000 7f23d17fa640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:43.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.497+0000 7f23d17fa640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:43.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.497+0000 7f23d17fa640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-25T18:22:43.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.498+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23c804ed10 con 0x7f23d40a98f0 2024-03-25T18:22:43.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.498+0000 7f23d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:43.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.498+0000 7f23d17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:43.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.749+0000 7f23db065640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:43.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.749+0000 7f23db065640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:43.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.749+0000 7f23db065640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:22:43.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.749+0000 7f23db065640 1 -- 172.21.15.122:0/2444483398 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23d413cb40 con 0x7f23d40a98f0 2024-03-25T18:22:43.770 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.770+0000 7f23d17fa640 1 -- 172.21.15.122:0/2444483398 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f23c8050e10 con 0x7f23d40a98f0 2024-03-25T18:22:43.770 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.770+0000 7f23d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:43.770 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.770+0000 7f23d17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:43.770 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:22:43.771 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:43.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.773+0000 7f23db065640 10 librados: watch_flush enter 2024-03-25T18:22:43.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.773+0000 7f23db065640 10 librados: watch_flush exit 2024-03-25T18:22:43.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.773+0000 7f23db065640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-25T18:22:43.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.773+0000 7f23db065640 10 client.4510.objecter successfully canceled tick 2024-03-25T18:22:43.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.773+0000 7f23db065640 1 -- 172.21.15.122:0/2444483398 >> v1:172.21.15.204:6800/676228005 conn(0x7f23b806deb0 legacy=0x7f23b8070370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:43.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.773+0000 7f23db065640 10 monclient: shutdown 2024-03-25T18:22:43.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.773+0000 7f23db065640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:43.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.773+0000 7f23db065640 1 -- 172.21.15.122:0/2444483398 >> v1:172.21.15.204:6789/0 conn(0x7f23d40a98f0 legacy=0x7f23d4130f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:43.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.773+0000 7f23db065640 1 -- 172.21.15.122:0/2444483398 shutdown_connections 2024-03-25T18:22:43.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.773+0000 7f23db065640 1 -- 172.21.15.122:0/2444483398 >> 172.21.15.122:0/2444483398 conn(0x7f23d40b7ea0 msgr2=0x7f23d414ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:43.775 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.773+0000 7f23db065640 1 -- 172.21.15.122:0/2444483398 shutdown_connections 2024-03-25T18:22:43.775 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.773+0000 7f23db065640 1 -- 172.21.15.122:0/2444483398 wait complete. 2024-03-25T18:22:43.775 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:43.773+0000 7f23db065640 1 librados: shutdown 2024-03-25T18:22:43.785 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:46.631 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:06:40.373 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:46.634 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:06:40.377 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:46.683 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:06:40.426 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:46.695 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:06:40.438 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:46.705 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:06:40.451 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:46.711 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:06:40.457 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:46.715 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:06:40.458 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:46.735 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:06:40.480 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:46.743 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:06:40.485 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:50.592 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:50.702 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.701+0000 7f91e791c640 10 monclient: get_monmap_and_config 2024-03-25T18:22:50.702 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.701+0000 7f91e791c640 10 monclient: build_initial_monmap 2024-03-25T18:22:50.703 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.701+0000 7f91e791c640 10 monclient: monmap: 2024-03-25T18:22:50.703 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:22:50.703 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:50.703 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:22:50.702706+0000 2024-03-25T18:22:50.703 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:22:50.702706+0000 2024-03-25T18:22:50.703 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:50.703 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:50.703 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:22:50.703 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:22:50.703 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:22:50.703 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:50.704 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.702+0000 7f91e791c640 1 Processor -- start 2024-03-25T18:22:50.704 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.702+0000 7f91e791c640 1 -- start start 2024-03-25T18:22:50.704 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.702+0000 7f91e791c640 10 monclient: init 2024-03-25T18:22:50.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.703+0000 7f91e791c640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:50.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.703+0000 7f91e791c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:50.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.703+0000 7f91e791c640 10 monclient(hunting): picked mon.noname-a con 0x7f91e00a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:22:50.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.703+0000 7f91e791c640 10 monclient(hunting): picked mon.noname-c con 0x7f91e013f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:22:50.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.703+0000 7f91e791c640 10 monclient(hunting): picked mon.noname-b con 0x7f91e01430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:22:50.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.703+0000 7f91e791c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e0142520 con 0x7f91e00a98f0 2024-03-25T18:22:50.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.703+0000 7f91e791c640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e00b28a0 con 0x7f91e013f710 2024-03-25T18:22:50.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.703+0000 7f91e791c640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e00b2a30 con 0x7f91e01430c0 2024-03-25T18:22:50.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.703+0000 7f91e791c640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:50.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.703+0000 7f91e791c640 10 monclient(hunting): authenticate will time out at 7693.969166s 2024-03-25T18:22:50.713 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.712+0000 7f91e56b8640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f91e00a98f0 0x7f91e00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50570/0 (socket says 172.21.15.122:50570) 2024-03-25T18:22:50.714 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.712+0000 7f91e56b8640 1 -- 172.21.15.122:0/1476338706 learned_addr learned my addr 172.21.15.122:0/1476338706 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:22:50.726 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.725+0000 7f91d7fff640 1 -- 172.21.15.122:0/1476338706 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153077807 0 0) 0x7f91e0142520 con 0x7f91e00a98f0 2024-03-25T18:22:50.726 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.725+0000 7f91d7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:50.726 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.725+0000 7f91d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:50.726 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.725+0000 7f91d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:50.726 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.725+0000 7f91d7fff640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:22:50.726 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.725+0000 7f91d7fff640 1 -- 172.21.15.122:0/1476338706 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91cc003520 con 0x7f91e00a98f0 2024-03-25T18:22:50.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.737+0000 7f91d7fff640 1 -- 172.21.15.122:0/1476338706 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2533971943 0 0) 0x7f91cc003520 con 0x7f91e00a98f0 2024-03-25T18:22:50.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.737+0000 7f91d7fff640 1 -- 172.21.15.122:0/1476338706 >> v1:172.21.15.122:6790/0 conn(0x7f91e01430c0 legacy=0x7f91e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:50.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.737+0000 7f91d7fff640 1 -- 172.21.15.122:0/1476338706 >> v1:172.21.15.204:6789/0 conn(0x7f91e013f710 legacy=0x7f91e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:50.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.737+0000 7f91d7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:50.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.737+0000 7f91d7fff640 1 monclient: found mon.noname-a 2024-03-25T18:22:50.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.737+0000 7f91d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:50.740 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.737+0000 7f91d7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:22:50.740 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.737+0000 7f91d7fff640 1 -- 172.21.15.122:0/1476338706 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e00b44c0 con 0x7f91e00a98f0 2024-03-25T18:22:50.740 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.737+0000 7f91d7fff640 10 monclient: _finish_auth 0 2024-03-25T18:22:50.740 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.737+0000 7f91d7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:50.740 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.737+0000 7f91d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:50.740 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.737+0000 7f91d7fff640 1 -- 172.21.15.122:0/1476338706 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f91dc001690 con 0x7f91e00a98f0 2024-03-25T18:22:50.740 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.737+0000 7f91d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:50.740 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.737+0000 7f91d7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:50.740 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.737+0000 7f91d7fff640 10 monclient: dump: 2024-03-25T18:22:50.740 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:50.740 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:50.740 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:50.740 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:50.740 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:50.740 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:50.740 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:50.741 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:50.741 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:50.741 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:50.741 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.737+0000 7f91e791c640 5 monclient: authenticate success, global_id 4515 2024-03-25T18:22:50.741 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.737+0000 7f91e791c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:50.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.818+0000 7f91d7fff640 1 -- 172.21.15.122:0/1476338706 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91dc0017f0 con 0x7f91e00a98f0 2024-03-25T18:22:50.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.818+0000 7f91d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:50.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.818+0000 7f91d7fff640 1 -- 172.21.15.122:0/1476338706 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f91dc001c00 con 0x7f91e00a98f0 2024-03-25T18:22:50.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.818+0000 7f91d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:50.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.818+0000 7f91d7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:50.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.818+0000 7f91d7fff640 10 monclient: dump: 2024-03-25T18:22:50.820 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:50.820 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:50.820 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:50.820 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:50.820 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:50.820 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:50.820 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:50.820 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:50.820 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:50.820 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:50.821 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.818+0000 7f91e791c640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:50.821 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.818+0000 7f91e791c640 4 set_mon_vals no callback set 2024-03-25T18:22:50.822 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.819+0000 7f91e791c640 10 monclient: shutdown 2024-03-25T18:22:50.822 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.819+0000 7f91e791c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:50.822 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.819+0000 7f91e791c640 1 -- 172.21.15.122:0/1476338706 >> v1:172.21.15.122:6789/0 conn(0x7f91e00a98f0 legacy=0x7f91e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:50.822 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.819+0000 7f91e791c640 1 -- 172.21.15.122:0/1476338706 shutdown_connections 2024-03-25T18:22:50.822 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.819+0000 7f91e791c640 1 -- 172.21.15.122:0/1476338706 >> 172.21.15.122:0/1476338706 conn(0x7f91e00b7ea0 msgr2=0x7f91e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:50.822 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.819+0000 7f91e791c640 1 -- 172.21.15.122:0/1476338706 shutdown_connections 2024-03-25T18:22:50.822 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.819+0000 7f91e791c640 1 -- 172.21.15.122:0/1476338706 wait complete. 2024-03-25T18:22:50.822 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.820+0000 7f91e791c640 10 monclient: build_initial_monmap 2024-03-25T18:22:50.822 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.820+0000 7f91e791c640 10 monclient: monmap: 2024-03-25T18:22:50.823 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:22:50.823 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:50.823 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:22:50.823 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:22:50.823 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:50.823 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:22:50.823 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:22:50.823 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:22:50.823 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:22:50.823 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:50.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.820+0000 7f91e791c640 1 librados: starting msgr at 2024-03-25T18:22:50.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.820+0000 7f91e791c640 1 librados: starting objecter 2024-03-25T18:22:50.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.821+0000 7f91e791c640 1 Processor -- start 2024-03-25T18:22:50.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.821+0000 7f91e791c640 1 -- start start 2024-03-25T18:22:50.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.821+0000 7f91e791c640 1 librados: setting wanted keys 2024-03-25T18:22:50.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.821+0000 7f91e791c640 1 librados: calling monclient init 2024-03-25T18:22:50.824 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.821+0000 7f91e791c640 10 monclient: init 2024-03-25T18:22:50.824 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.821+0000 7f91e791c640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:50.824 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.821+0000 7f91e791c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:50.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.821+0000 7f91e791c640 10 monclient(hunting): picked mon.noname-c con 0x7f91e00a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:22:50.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.821+0000 7f91e791c640 10 monclient(hunting): picked mon.noname-a con 0x7f91e013f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:22:50.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.821+0000 7f91e791c640 10 monclient(hunting): picked mon.noname-b con 0x7f91e01430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:22:50.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.821+0000 7f91e791c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e0138c70 con 0x7f91e013f710 2024-03-25T18:22:50.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.821+0000 7f91e791c640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e01367d0 con 0x7f91e01430c0 2024-03-25T18:22:50.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.821+0000 7f91e791c640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e0136960 con 0x7f91e00a98f0 2024-03-25T18:22:50.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.821+0000 7f91e791c640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:50.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.821+0000 7f91e791c640 10 monclient(hunting): authenticate will time out at 7694.087866s 2024-03-25T18:22:50.840 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.839+0000 7f91e4eb7640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f91e013f710 0x7f91e0132f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:50612/0 (socket says 172.21.15.122:50612) 2024-03-25T18:22:50.840 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.839+0000 7f91e4eb7640 1 -- 172.21.15.122:0/2228344847 learned_addr learned my addr 172.21.15.122:0/2228344847 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:22:50.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.840+0000 7f91d5ffb640 10 client.?.objecter ms_handle_connect 0x7f91e013f710 2024-03-25T18:22:50.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.840+0000 7f91d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:50.847 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.845+0000 7f91d5ffb640 1 -- 172.21.15.122:0/2228344847 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2143026852 0 0) 0x7f91e0138c70 con 0x7f91e013f710 2024-03-25T18:22:50.847 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.845+0000 7f91d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:50.847 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.845+0000 7f91d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:50.847 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.845+0000 7f91d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:50.847 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.845+0000 7f91d5ffb640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:22:50.847 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.845+0000 7f91d5ffb640 1 -- 172.21.15.122:0/2228344847 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c4004eb0 con 0x7f91e013f710 2024-03-25T18:22:50.854 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.853+0000 7f91d5ffb640 1 -- 172.21.15.122:0/2228344847 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3818907896 0 0) 0x7f91c4004eb0 con 0x7f91e013f710 2024-03-25T18:22:50.854 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.853+0000 7f91d5ffb640 1 -- 172.21.15.122:0/2228344847 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f91e0138c70 con 0x7f91e013f710 2024-03-25T18:22:50.854 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.853+0000 7f91d5ffb640 1 -- 172.21.15.122:0/2228344847 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f91d00019d0 con 0x7f91e013f710 2024-03-25T18:22:50.854 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.853+0000 7f91d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:50.854 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.853+0000 7f91d5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:50.854 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.853+0000 7f91d5ffb640 10 monclient(hunting): dump: 2024-03-25T18:22:50.854 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:50.855 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:50.855 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:50.855 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:50.855 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:50.855 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:50.855 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:50.855 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:50.855 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:50.855 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:50.855 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.853+0000 7f91d5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:50.863 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.862+0000 7f91d5ffb640 10 client.?.objecter ms_handle_connect 0x7f91e01430c0 2024-03-25T18:22:50.863 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.862+0000 7f91d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:50.865 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91d5ffb640 1 -- 172.21.15.122:0/2228344847 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 438160676 0 0) 0x7f91e0138c70 con 0x7f91e013f710 2024-03-25T18:22:50.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91d5ffb640 1 -- 172.21.15.122:0/2228344847 >> v1:172.21.15.122:6790/0 conn(0x7f91e00a98f0 legacy=0x7f91e0137610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:50.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91d5ffb640 1 -- 172.21.15.122:0/2228344847 >> v1:172.21.15.204:6789/0 conn(0x7f91e01430c0 legacy=0x7f91e01e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:50.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91d5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:22:50.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91d5ffb640 1 monclient: found mon.a 2024-03-25T18:22:50.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:50.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91d5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:22:50.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91d5ffb640 1 -- 172.21.15.122:0/2228344847 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e0139630 con 0x7f91e013f710 2024-03-25T18:22:50.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91d5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:22:50.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91d5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:22:50.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:50.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91e791c640 5 monclient: authenticate success, global_id 4518 2024-03-25T18:22:50.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91e791c640 10 monclient: _renew_subs 2024-03-25T18:22:50.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91e791c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:22:50.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91e791c640 1 -- 172.21.15.122:0/2228344847 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e0139880 con 0x7f91e013f710 2024-03-25T18:22:50.867 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91e791c640 10 client.4518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:50.867 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91e791c640 10 monclient: _renew_subs 2024-03-25T18:22:50.867 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91e791c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:22:50.867 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91e791c640 1 -- 172.21.15.122:0/2228344847 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e01337f0 con 0x7f91e013f710 2024-03-25T18:22:50.867 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.864+0000 7f91e791c640 1 librados: init done 2024-03-25T18:22:50.867 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.865+0000 7f91e791c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:50.868 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.865+0000 7f91e791c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:50.868 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.865+0000 7f91e791c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:22:50.868 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:50.865+0000 7f91e791c640 1 -- 172.21.15.122:0/2228344847 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e00627b0 con 0x7f91e013f710 2024-03-25T18:22:51.126 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.125+0000 7f91d5ffb640 1 -- 172.21.15.122:0/2228344847 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f91d0001cd0 con 0x7f91e013f710 2024-03-25T18:22:51.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.125+0000 7f91d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:51.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.125+0000 7f91d5ffb640 1 -- 172.21.15.122:0/2228344847 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f91d0003a60 con 0x7f91e013f710 2024-03-25T18:22:51.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.125+0000 7f91d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:51.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.125+0000 7f91d5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:51.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.125+0000 7f91d5ffb640 10 monclient: dump: 2024-03-25T18:22:51.127 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:51.127 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:51.127 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:51.127 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:51.127 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:51.127 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:51.127 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:51.127 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:51.127 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:51.127 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:51.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.125+0000 7f91d77fe640 4 set_mon_vals no callback set 2024-03-25T18:22:51.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.125+0000 7f91d5ffb640 1 -- 172.21.15.122:0/2228344847 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f91d0003c80 con 0x7f91e013f710 2024-03-25T18:22:51.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.126+0000 7f91e56b8640 10 monclient: get_auth_request con 0x7f91c406dbf0 auth_method 0 2024-03-25T18:22:51.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.126+0000 7f91d5ffb640 10 client.4518.objecter ms_handle_connect 0x7f91c406dbf0 2024-03-25T18:22:51.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.130+0000 7f91d5ffb640 1 -- 172.21.15.122:0/2228344847 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f91d00548a0 con 0x7f91e013f710 2024-03-25T18:22:51.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.130+0000 7f91d5ffb640 10 client.4518.objecter ms_dispatch 0x7f91e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:51.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.130+0000 7f91d5ffb640 3 client.4518.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:51.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.130+0000 7f91d5ffb640 3 client.4518.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:51.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.130+0000 7f91d5ffb640 20 client.4518.objecter dump_active .. 0 homeless 2024-03-25T18:22:51.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.130+0000 7f91d5ffb640 1 -- 172.21.15.122:0/2228344847 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f91d00503b0 con 0x7f91e013f710 2024-03-25T18:22:51.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.130+0000 7f91d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:51.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.130+0000 7f91d5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:51.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.359+0000 7f91e791c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:51.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.359+0000 7f91e791c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:51.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.359+0000 7f91e791c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:22:51.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.359+0000 7f91e791c640 1 -- 172.21.15.122:0/2228344847 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f91e0069f40 con 0x7f91e013f710 2024-03-25T18:22:51.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.382+0000 7f91d5ffb640 1 -- 172.21.15.122:0/2228344847 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f91d00522e0 con 0x7f91e013f710 2024-03-25T18:22:51.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.382+0000 7f91d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:51.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.382+0000 7f91d5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:51.384 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:22:51.384 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:51.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.385+0000 7f91e791c640 10 librados: watch_flush enter 2024-03-25T18:22:51.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.385+0000 7f91e791c640 10 librados: watch_flush exit 2024-03-25T18:22:51.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.385+0000 7f91e791c640 20 client.4518.objecter shutdown clearing up homeless session... 2024-03-25T18:22:51.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.385+0000 7f91e791c640 10 client.4518.objecter successfully canceled tick 2024-03-25T18:22:51.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.385+0000 7f91e791c640 1 -- 172.21.15.122:0/2228344847 >> v1:172.21.15.204:6800/676228005 conn(0x7f91c406dbf0 legacy=0x7f91c40700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:51.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.385+0000 7f91e791c640 10 monclient: shutdown 2024-03-25T18:22:51.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.385+0000 7f91e791c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:51.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.385+0000 7f91e791c640 1 -- 172.21.15.122:0/2228344847 >> v1:172.21.15.122:6789/0 conn(0x7f91e013f710 legacy=0x7f91e0132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:51.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.385+0000 7f91e791c640 1 -- 172.21.15.122:0/2228344847 shutdown_connections 2024-03-25T18:22:51.388 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.385+0000 7f91e791c640 1 -- 172.21.15.122:0/2228344847 >> 172.21.15.122:0/2228344847 conn(0x7f91e00b7ea0 msgr2=0x7f91e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:51.388 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.385+0000 7f91e791c640 1 -- 172.21.15.122:0/2228344847 shutdown_connections 2024-03-25T18:22:51.388 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.386+0000 7f91e791c640 1 -- 172.21.15.122:0/2228344847 wait complete. 2024-03-25T18:22:51.388 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:51.386+0000 7f91e791c640 1 librados: shutdown 2024-03-25T18:22:51.399 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:58.206 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:58.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.317+0000 7fd4793cf640 10 monclient: get_monmap_and_config 2024-03-25T18:22:58.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.317+0000 7fd4793cf640 10 monclient: build_initial_monmap 2024-03-25T18:22:58.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.317+0000 7fd4793cf640 10 monclient: monmap: 2024-03-25T18:22:58.319 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:22:58.319 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:58.319 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:22:58.318474+0000 2024-03-25T18:22:58.319 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:22:58.318474+0000 2024-03-25T18:22:58.319 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:58.319 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:58.319 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:22:58.319 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:22:58.319 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:22:58.319 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:58.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.318+0000 7fd4793cf640 1 Processor -- start 2024-03-25T18:22:58.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.318+0000 7fd4793cf640 1 -- start start 2024-03-25T18:22:58.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.318+0000 7fd4793cf640 10 monclient: init 2024-03-25T18:22:58.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.318+0000 7fd4793cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:58.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.318+0000 7fd4793cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:58.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.318+0000 7fd4793cf640 10 monclient(hunting): picked mon.noname-b con 0x7fd4740a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:22:58.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.318+0000 7fd4793cf640 10 monclient(hunting): picked mon.noname-c con 0x7fd47413f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:22:58.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.318+0000 7fd4793cf640 10 monclient(hunting): picked mon.noname-a con 0x7fd4741430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:22:58.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.318+0000 7fd4793cf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd474142520 con 0x7fd4741430c0 2024-03-25T18:22:58.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.318+0000 7fd4793cf640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4740b28a0 con 0x7fd47413f710 2024-03-25T18:22:58.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.318+0000 7fd4793cf640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4740b2a30 con 0x7fd4740a98f0 2024-03-25T18:22:58.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.318+0000 7fd4793cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:58.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.318+0000 7fd4793cf640 10 monclient(hunting): authenticate will time out at 7701.584955s 2024-03-25T18:22:58.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.321+0000 7fd4737fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd4741430c0 0x7fd47414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:55342/0 (socket says 172.21.15.122:55342) 2024-03-25T18:22:58.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.321+0000 7fd4737fe640 1 -- 172.21.15.122:0/1857986909 learned_addr learned my addr 172.21.15.122:0/1857986909 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:22:58.332 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.331+0000 7fd4727fc640 1 -- 172.21.15.122:0/1857986909 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559192924 0 0) 0x7fd474142520 con 0x7fd4741430c0 2024-03-25T18:22:58.332 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.331+0000 7fd4727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:58.332 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.331+0000 7fd4727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:58.332 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.331+0000 7fd4727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:58.332 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.331+0000 7fd4727fc640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:22:58.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.331+0000 7fd4727fc640 1 -- 172.21.15.122:0/1857986909 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd460003520 con 0x7fd4741430c0 2024-03-25T18:22:58.346 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.344+0000 7fd4727fc640 1 -- 172.21.15.122:0/1857986909 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417033266 0 0) 0x7fd4740b2a30 con 0x7fd4740a98f0 2024-03-25T18:22:58.346 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.345+0000 7fd4727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:58.346 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.345+0000 7fd4727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:58.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.345+0000 7fd4727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:58.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.345+0000 7fd4727fc640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:22:58.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.345+0000 7fd4727fc640 1 -- 172.21.15.122:0/1857986909 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd474142520 con 0x7fd4740a98f0 2024-03-25T18:22:58.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.345+0000 7fd4727fc640 1 -- 172.21.15.122:0/1857986909 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2038064443 0 0) 0x7fd4740b28a0 con 0x7fd47413f710 2024-03-25T18:22:58.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.345+0000 7fd4727fc640 10 monclient(hunting): _negotiate 2024-03-25T18:22:58.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.345+0000 7fd4727fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:58.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.345+0000 7fd4727fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:58.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.345+0000 7fd4727fc640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:22:58.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.345+0000 7fd4727fc640 1 -- 172.21.15.122:0/1857986909 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4740b2a30 con 0x7fd47413f710 2024-03-25T18:22:58.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4727fc640 1 -- 172.21.15.122:0/1857986909 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2378386363 0 0) 0x7fd460003520 con 0x7fd4741430c0 2024-03-25T18:22:58.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4727fc640 1 -- 172.21.15.122:0/1857986909 >> v1:172.21.15.122:6790/0 conn(0x7fd4740a98f0 legacy=0x7fd4740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:58.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4727fc640 1 -- 172.21.15.122:0/1857986909 >> v1:172.21.15.204:6789/0 conn(0x7fd47413f710 legacy=0x7fd4741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:58.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4727fc640 10 monclient: _finish_hunting 0 2024-03-25T18:22:58.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4727fc640 1 monclient: found mon.noname-a 2024-03-25T18:22:58.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:58.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4727fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:22:58.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4727fc640 1 -- 172.21.15.122:0/1857986909 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4740b44c0 con 0x7fd4741430c0 2024-03-25T18:22:58.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4727fc640 10 monclient: _finish_auth 0 2024-03-25T18:22:58.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4727fc640 10 monclient: _check_auth_tickets 2024-03-25T18:22:58.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:58.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4727fc640 1 -- 172.21.15.122:0/1857986909 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd46c001a10 con 0x7fd4741430c0 2024-03-25T18:22:58.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4793cf640 5 monclient: authenticate success, global_id 4521 2024-03-25T18:22:58.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4793cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:58.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:58.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:58.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4727fc640 10 monclient: dump: 2024-03-25T18:22:58.350 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:58.351 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:58.351 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:58.351 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:58.351 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:58.351 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:58.351 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:58.351 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:58.351 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:58.351 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:58.351 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.347+0000 7fd4793cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.360+0000 7fd4727fc640 1 -- 172.21.15.122:0/1857986909 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd46c001b70 con 0x7fd4741430c0 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.360+0000 7fd4727fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.360+0000 7fd4727fc640 1 -- 172.21.15.122:0/1857986909 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd46c001f80 con 0x7fd4741430c0 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.360+0000 7fd4793cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.360+0000 7fd4793cf640 4 set_mon_vals no callback set 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.360+0000 7fd4727fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.360+0000 7fd4727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.360+0000 7fd4727fc640 10 monclient: dump: 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:58.362 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:58.363 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:58.363 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:58.363 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:58.363 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:58.363 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:58.363 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:58.363 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:58.363 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:58.364 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.361+0000 7fd4793cf640 10 monclient: shutdown 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.361+0000 7fd4793cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.361+0000 7fd4793cf640 1 -- 172.21.15.122:0/1857986909 >> v1:172.21.15.122:6789/0 conn(0x7fd4741430c0 legacy=0x7fd47414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.361+0000 7fd4793cf640 1 -- 172.21.15.122:0/1857986909 shutdown_connections 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.361+0000 7fd4793cf640 1 -- 172.21.15.122:0/1857986909 >> 172.21.15.122:0/1857986909 conn(0x7fd4740b7ea0 msgr2=0x7fd4740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.361+0000 7fd4793cf640 1 -- 172.21.15.122:0/1857986909 shutdown_connections 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.362+0000 7fd4793cf640 1 -- 172.21.15.122:0/1857986909 wait complete. 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.363+0000 7fd4793cf640 10 monclient: build_initial_monmap 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.363+0000 7fd4793cf640 10 monclient: monmap: 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:22:58.365 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:22:58.366 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:58.366 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:22:58.366 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:22:58.366 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:22:58.366 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:22:58.366 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:58.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.363+0000 7fd4793cf640 1 librados: starting msgr at 2024-03-25T18:22:58.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.363+0000 7fd4793cf640 1 librados: starting objecter 2024-03-25T18:22:58.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.363+0000 7fd4793cf640 1 Processor -- start 2024-03-25T18:22:58.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.363+0000 7fd4793cf640 1 -- start start 2024-03-25T18:22:58.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.363+0000 7fd4793cf640 1 librados: setting wanted keys 2024-03-25T18:22:58.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.363+0000 7fd4793cf640 1 librados: calling monclient init 2024-03-25T18:22:58.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.363+0000 7fd4793cf640 10 monclient: init 2024-03-25T18:22:58.367 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.364+0000 7fd4793cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:58.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.364+0000 7fd4793cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:58.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.364+0000 7fd4793cf640 10 monclient(hunting): picked mon.noname-c con 0x7fd4740a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:22:58.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.364+0000 7fd4793cf640 10 monclient(hunting): picked mon.noname-a con 0x7fd47413f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:22:58.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.364+0000 7fd4793cf640 10 monclient(hunting): picked mon.noname-b con 0x7fd4741430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:22:58.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.364+0000 7fd4793cf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4741d7df0 con 0x7fd47413f710 2024-03-25T18:22:58.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.364+0000 7fd4793cf640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4741d7f80 con 0x7fd4741430c0 2024-03-25T18:22:58.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.364+0000 7fd4793cf640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd47413b5c0 con 0x7fd4740a98f0 2024-03-25T18:22:58.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.364+0000 7fd4793cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:58.368 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.364+0000 7fd4793cf640 10 monclient(hunting): authenticate will time out at 7701.630596s 2024-03-25T18:22:58.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.370+0000 7fd46a7fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fd47413f710 0x7fd474133000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:55402/0 (socket says 172.21.15.122:55402) 2024-03-25T18:22:58.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.370+0000 7fd46a7fc640 1 -- 172.21.15.122:0/3597076022 learned_addr learned my addr 172.21.15.122:0/3597076022 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:22:58.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.376+0000 7fd46bfff640 10 client.?.objecter ms_handle_connect 0x7fd47413f710 2024-03-25T18:22:58.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.376+0000 7fd46bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:58.378 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.377+0000 7fd46bfff640 10 client.?.objecter ms_handle_connect 0x7fd4741430c0 2024-03-25T18:22:58.379 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.377+0000 7fd46bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:58.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.384+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464469233 0 0) 0x7fd4741d7df0 con 0x7fd47413f710 2024-03-25T18:22:58.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.384+0000 7fd46bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:58.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.384+0000 7fd46bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:58.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.384+0000 7fd46bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:58.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.384+0000 7fd46bfff640 10 monclient(hunting): my global_id is 4524 2024-03-25T18:22:58.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.384+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd454004eb0 con 0x7fd47413f710 2024-03-25T18:22:58.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.393+0000 7fd46bfff640 10 client.?.objecter ms_handle_connect 0x7fd4740a98f0 2024-03-25T18:22:58.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.393+0000 7fd46bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:58.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.394+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820575138 0 0) 0x7fd4741d7f80 con 0x7fd4741430c0 2024-03-25T18:22:58.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.394+0000 7fd46bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:58.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.394+0000 7fd46bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:58.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.394+0000 7fd46bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:58.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.394+0000 7fd46bfff640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:22:58.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.394+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4741d7df0 con 0x7fd4741430c0 2024-03-25T18:22:58.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.395+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3066951956 0 0) 0x7fd454004eb0 con 0x7fd47413f710 2024-03-25T18:22:58.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.395+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd4741d7f80 con 0x7fd47413f710 2024-03-25T18:22:58.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.395+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd45c001a10 con 0x7fd47413f710 2024-03-25T18:22:58.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.395+0000 7fd46bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:58.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.395+0000 7fd46bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:58.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.395+0000 7fd46bfff640 10 monclient(hunting): dump: 2024-03-25T18:22:58.397 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:58.397 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:58.397 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:58.397 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:58.397 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:58.397 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:58.398 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:58.398 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:58.398 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:58.398 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:58.398 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.395+0000 7fd46bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:58.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.406+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 609378894 0 0) 0x7fd47413b5c0 con 0x7fd4740a98f0 2024-03-25T18:22:58.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.406+0000 7fd46bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:58.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.406+0000 7fd46bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:58.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.406+0000 7fd46bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:58.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.406+0000 7fd46bfff640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:22:58.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.406+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd454004eb0 con 0x7fd4740a98f0 2024-03-25T18:22:58.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.418+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4059123949 0 0) 0x7fd4741d7df0 con 0x7fd4741430c0 2024-03-25T18:22:58.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.418+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd47413b5c0 con 0x7fd4741430c0 2024-03-25T18:22:58.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.418+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd46c001e80 con 0x7fd4741430c0 2024-03-25T18:22:58.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.418+0000 7fd46bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:58.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.418+0000 7fd46bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:58.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.418+0000 7fd46bfff640 10 monclient(hunting): dump: 2024-03-25T18:22:58.420 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:58.420 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:58.420 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:58.420 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:58.420 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:58.420 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:58.420 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:58.420 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:58.420 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:58.420 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 159005098 0 0) 0x7fd47413b5c0 con 0x7fd4741430c0 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 >> v1:172.21.15.122:6790/0 conn(0x7fd4740a98f0 legacy=0x7fd4741d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 >> v1:172.21.15.122:6789/0 conn(0x7fd47413f710 legacy=0x7fd474133000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd46bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd46bfff640 1 monclient: found mon.b 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd46bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd46bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47413b750 con 0x7fd4741430c0 2024-03-25T18:22:58.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd46bfff640 10 monclient: _finish_auth 0 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd46bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd46bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd4793cf640 5 monclient: authenticate success, global_id 4516 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd4793cf640 10 monclient: _renew_subs 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd4793cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd4793cf640 1 -- 172.21.15.122:0/3597076022 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd474133710 con 0x7fd4741430c0 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd4793cf640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd4793cf640 10 monclient: _renew_subs 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd4793cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd4793cf640 1 -- 172.21.15.122:0/3597076022 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4741357a0 con 0x7fd4741430c0 2024-03-25T18:22:58.431 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.428+0000 7fd4793cf640 1 librados: init done 2024-03-25T18:22:58.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.429+0000 7fd4793cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:58.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.429+0000 7fd4793cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:58.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.429+0000 7fd4793cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:22:58.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.429+0000 7fd4793cf640 1 -- 172.21.15.122:0/3597076022 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd474138fd0 con 0x7fd4741430c0 2024-03-25T18:22:58.716 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.714+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd46c001fe0 con 0x7fd4741430c0 2024-03-25T18:22:58.716 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.714+0000 7fd46bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:58.716 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.714+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fd46c004e10 con 0x7fd4741430c0 2024-03-25T18:22:58.716 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.714+0000 7fd46bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:58.716 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.714+0000 7fd46bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:58.716 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.714+0000 7fd46bfff640 10 monclient: dump: 2024-03-25T18:22:58.716 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:22:58.716 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:22:58.716 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:58.717 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:22:58.717 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:58.717 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:22:58.717 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:22:58.717 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:22:58.717 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:22:58.717 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:22:58.717 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.714+0000 7fd471ffb640 4 set_mon_vals no callback set 2024-03-25T18:22:58.718 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.715+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fd46c01ade0 con 0x7fd4741430c0 2024-03-25T18:22:58.718 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.716+0000 7fd472ffd640 10 monclient: get_auth_request con 0x7fd45406df20 auth_method 0 2024-03-25T18:22:58.718 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.716+0000 7fd46bfff640 10 client.4516.objecter ms_handle_connect 0x7fd45406df20 2024-03-25T18:22:58.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.722+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fd46c054960 con 0x7fd4741430c0 2024-03-25T18:22:58.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.722+0000 7fd46bfff640 10 client.4516.objecter ms_dispatch 0x7fd474000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:58.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.722+0000 7fd46bfff640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:58.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.722+0000 7fd46bfff640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:58.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.722+0000 7fd46bfff640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-25T18:22:58.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.723+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd46c050470 con 0x7fd4741430c0 2024-03-25T18:22:58.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.723+0000 7fd46bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:58.724 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.723+0000 7fd46bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:58.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.972+0000 7fd4793cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:58.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.972+0000 7fd4793cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:58.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.972+0000 7fd4793cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:22:58.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:58.972+0000 7fd4793cf640 1 -- 172.21.15.122:0/3597076022 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd474074cc0 con 0x7fd4741430c0 2024-03-25T18:22:59.026 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:59.025+0000 7fd46bfff640 1 -- 172.21.15.122:0/3597076022 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fd46c0523a0 con 0x7fd4741430c0 2024-03-25T18:22:59.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:59.025+0000 7fd46bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:59.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:59.025+0000 7fd46bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:59.027 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:22:59.027 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:59.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:59.027+0000 7fd4793cf640 10 librados: watch_flush enter 2024-03-25T18:22:59.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:59.027+0000 7fd4793cf640 10 librados: watch_flush exit 2024-03-25T18:22:59.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:59.027+0000 7fd4793cf640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-25T18:22:59.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:59.027+0000 7fd4793cf640 10 client.4516.objecter successfully canceled tick 2024-03-25T18:22:59.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:59.027+0000 7fd4793cf640 1 -- 172.21.15.122:0/3597076022 >> v1:172.21.15.204:6800/676228005 conn(0x7fd45406df20 legacy=0x7fd4540703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:59.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:59.027+0000 7fd4793cf640 10 monclient: shutdown 2024-03-25T18:22:59.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:59.027+0000 7fd4793cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:59.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:59.027+0000 7fd4793cf640 1 -- 172.21.15.122:0/3597076022 >> v1:172.21.15.204:6789/0 conn(0x7fd4741430c0 legacy=0x7fd474135090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:59.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:59.027+0000 7fd4793cf640 1 -- 172.21.15.122:0/3597076022 shutdown_connections 2024-03-25T18:22:59.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:59.027+0000 7fd4793cf640 1 -- 172.21.15.122:0/3597076022 >> 172.21.15.122:0/3597076022 conn(0x7fd4740b7ea0 msgr2=0x7fd474074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:59.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:59.028+0000 7fd4793cf640 1 -- 172.21.15.122:0/3597076022 shutdown_connections 2024-03-25T18:22:59.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:59.028+0000 7fd4793cf640 1 -- 172.21.15.122:0/3597076022 wait complete. 2024-03-25T18:22:59.030 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:22:59.028+0000 7fd4793cf640 1 librados: shutdown 2024-03-25T18:22:59.041 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:05.847 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:05.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.958+0000 7fad6348d640 10 monclient: get_monmap_and_config 2024-03-25T18:23:05.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.958+0000 7fad6348d640 10 monclient: build_initial_monmap 2024-03-25T18:23:05.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.958+0000 7fad6348d640 10 monclient: monmap: 2024-03-25T18:23:05.960 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:23:05.960 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:05.960 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:23:05.960050+0000 2024-03-25T18:23:05.960 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:23:05.960050+0000 2024-03-25T18:23:05.960 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:05.960 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:05.960 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:23:05.961 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:23:05.961 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:23:05.961 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:05.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.959+0000 7fad6348d640 1 Processor -- start 2024-03-25T18:23:05.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.959+0000 7fad6348d640 1 -- start start 2024-03-25T18:23:05.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.959+0000 7fad6348d640 10 monclient: init 2024-03-25T18:23:05.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.960+0000 7fad6348d640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:05.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.960+0000 7fad6348d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:05.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.960+0000 7fad6348d640 10 monclient(hunting): picked mon.noname-b con 0x7fad5c0a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:23:05.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.960+0000 7fad6348d640 10 monclient(hunting): picked mon.noname-c con 0x7fad5c13f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:23:05.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.960+0000 7fad6348d640 10 monclient(hunting): picked mon.noname-a con 0x7fad5c1430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:23:05.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.960+0000 7fad6348d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5c142520 con 0x7fad5c1430c0 2024-03-25T18:23:05.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.960+0000 7fad6348d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5c0b28a0 con 0x7fad5c13f710 2024-03-25T18:23:05.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.960+0000 7fad6348d640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5c0b2a30 con 0x7fad5c0a98f0 2024-03-25T18:23:05.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.960+0000 7fad6348d640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:05.963 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.960+0000 7fad6348d640 10 monclient(hunting): authenticate will time out at 7709.226573s 2024-03-25T18:23:05.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.969+0000 7fad60a28640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fad5c13f710 0x7fad5c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:55710/0 (socket says 172.21.15.122:55710) 2024-03-25T18:23:05.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.969+0000 7fad60a28640 1 -- 172.21.15.122:0/1868448456 learned_addr learned my addr 172.21.15.122:0/1868448456 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:23:05.993 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.991+0000 7fad53fff640 1 -- 172.21.15.122:0/1868448456 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2302305302 0 0) 0x7fad5c142520 con 0x7fad5c1430c0 2024-03-25T18:23:05.993 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.991+0000 7fad53fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:05.993 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.991+0000 7fad53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:05.993 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.991+0000 7fad53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:05.993 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.991+0000 7fad53fff640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:23:05.993 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.991+0000 7fad53fff640 1 -- 172.21.15.122:0/1868448456 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad48003520 con 0x7fad5c1430c0 2024-03-25T18:23:05.993 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.992+0000 7fad53fff640 1 -- 172.21.15.122:0/1868448456 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 881008097 0 0) 0x7fad5c0b28a0 con 0x7fad5c13f710 2024-03-25T18:23:05.994 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.992+0000 7fad53fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:05.994 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.992+0000 7fad53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:05.994 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.992+0000 7fad53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:05.994 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.992+0000 7fad53fff640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:23:05.994 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:05.992+0000 7fad53fff640 1 -- 172.21.15.122:0/1868448456 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad5c142520 con 0x7fad5c13f710 2024-03-25T18:23:06.001 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.000+0000 7fad53fff640 1 -- 172.21.15.122:0/1868448456 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640961369 0 0) 0x7fad5c0b2a30 con 0x7fad5c0a98f0 2024-03-25T18:23:06.001 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.000+0000 7fad53fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:06.001 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.000+0000 7fad53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:06.001 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.000+0000 7fad53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:06.002 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.000+0000 7fad53fff640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:23:06.002 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.000+0000 7fad53fff640 1 -- 172.21.15.122:0/1868448456 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad5c0b28a0 con 0x7fad5c0a98f0 2024-03-25T18:23:06.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.010+0000 7fad53fff640 1 -- 172.21.15.122:0/1868448456 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2914943966 0 0) 0x7fad5c0b28a0 con 0x7fad5c0a98f0 2024-03-25T18:23:06.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.010+0000 7fad53fff640 1 -- 172.21.15.122:0/1868448456 >> v1:172.21.15.204:6789/0 conn(0x7fad5c13f710 legacy=0x7fad5c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:06.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.010+0000 7fad53fff640 1 -- 172.21.15.122:0/1868448456 >> v1:172.21.15.122:6789/0 conn(0x7fad5c1430c0 legacy=0x7fad5c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:06.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.010+0000 7fad53fff640 10 monclient: _finish_hunting 0 2024-03-25T18:23:06.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.010+0000 7fad53fff640 1 monclient: found mon.noname-b 2024-03-25T18:23:06.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.010+0000 7fad53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:06.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.010+0000 7fad53fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:23:06.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.010+0000 7fad53fff640 1 -- 172.21.15.122:0/1868448456 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5c0b44c0 con 0x7fad5c0a98f0 2024-03-25T18:23:06.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.010+0000 7fad53fff640 10 monclient: _finish_auth 0 2024-03-25T18:23:06.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.010+0000 7fad53fff640 10 monclient: _check_auth_tickets 2024-03-25T18:23:06.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.010+0000 7fad53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:06.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.010+0000 7fad6348d640 5 monclient: authenticate success, global_id 4490 2024-03-25T18:23:06.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.010+0000 7fad6348d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:06.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.018+0000 7fad53fff640 1 -- 172.21.15.122:0/1868448456 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fad580014f0 con 0x7fad5c0a98f0 2024-03-25T18:23:06.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.018+0000 7fad53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:06.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.018+0000 7fad53fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:06.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.018+0000 7fad53fff640 10 monclient: dump: 2024-03-25T18:23:06.020 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:06.020 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:06.021 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:06.021 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:06.021 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:06.021 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:06.021 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:06.021 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:06.021 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:06.021 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:06.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.018+0000 7fad53fff640 1 -- 172.21.15.122:0/1868448456 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad580017c0 con 0x7fad5c0a98f0 2024-03-25T18:23:06.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.018+0000 7fad6348d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:06.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.018+0000 7fad53fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:06.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.019+0000 7fad53fff640 1 -- 172.21.15.122:0/1868448456 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fad58001b50 con 0x7fad5c0a98f0 2024-03-25T18:23:06.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.019+0000 7fad53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:06.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.019+0000 7fad53fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:06.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.019+0000 7fad53fff640 10 monclient: dump: 2024-03-25T18:23:06.021 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:06.022 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:06.022 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:06.022 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:06.022 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:06.022 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:06.022 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:06.022 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:06.022 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:06.022 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:06.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.019+0000 7fad6348d640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:06.022 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.019+0000 7fad6348d640 4 set_mon_vals no callback set 2024-03-25T18:23:06.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.019+0000 7fad6348d640 10 monclient: shutdown 2024-03-25T18:23:06.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.019+0000 7fad6348d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:06.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.019+0000 7fad6348d640 1 -- 172.21.15.122:0/1868448456 >> v1:172.21.15.122:6790/0 conn(0x7fad5c0a98f0 legacy=0x7fad5c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:06.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.020+0000 7fad6348d640 1 -- 172.21.15.122:0/1868448456 shutdown_connections 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.020+0000 7fad6348d640 1 -- 172.21.15.122:0/1868448456 >> 172.21.15.122:0/1868448456 conn(0x7fad5c0b7ea0 msgr2=0x7fad5c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.020+0000 7fad6348d640 1 -- 172.21.15.122:0/1868448456 shutdown_connections 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.020+0000 7fad6348d640 1 -- 172.21.15.122:0/1868448456 wait complete. 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.020+0000 7fad6348d640 10 monclient: build_initial_monmap 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.020+0000 7fad6348d640 10 monclient: monmap: 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:23:06.025 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:23:06.026 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:23:06.026 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:06.026 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.021+0000 7fad6348d640 1 librados: starting msgr at 2024-03-25T18:23:06.026 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.021+0000 7fad6348d640 1 librados: starting objecter 2024-03-25T18:23:06.026 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.021+0000 7fad6348d640 1 Processor -- start 2024-03-25T18:23:06.026 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.021+0000 7fad6348d640 1 -- start start 2024-03-25T18:23:06.026 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.021+0000 7fad6348d640 1 librados: setting wanted keys 2024-03-25T18:23:06.026 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.021+0000 7fad6348d640 1 librados: calling monclient init 2024-03-25T18:23:06.026 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.021+0000 7fad6348d640 10 monclient: init 2024-03-25T18:23:06.026 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.021+0000 7fad6348d640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:06.026 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.021+0000 7fad6348d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:06.026 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.021+0000 7fad6348d640 10 monclient(hunting): picked mon.noname-c con 0x7fad5c0a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:23:06.026 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.021+0000 7fad6348d640 10 monclient(hunting): picked mon.noname-b con 0x7fad5c13f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:23:06.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.021+0000 7fad6348d640 10 monclient(hunting): picked mon.noname-a con 0x7fad5c1430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:23:06.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.021+0000 7fad6348d640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5c138fc0 con 0x7fad5c1430c0 2024-03-25T18:23:06.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.021+0000 7fad6348d640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5c1d7590 con 0x7fad5c13f710 2024-03-25T18:23:06.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.021+0000 7fad6348d640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad5c1d51f0 con 0x7fad5c0a98f0 2024-03-25T18:23:06.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.021+0000 7fad6348d640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:06.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.021+0000 7fad6348d640 10 monclient(hunting): authenticate will time out at 7709.287888s 2024-03-25T18:23:06.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.032+0000 7fad61229640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7fad5c0a98f0 0x7fad5c1388b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:37170/0 (socket says 172.21.15.122:37170) 2024-03-25T18:23:06.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.032+0000 7fad61229640 1 -- 172.21.15.122:0/3332575313 learned_addr learned my addr 172.21.15.122:0/3332575313 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:23:06.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.033+0000 7fad51ffb640 10 client.?.objecter ms_handle_connect 0x7fad5c0a98f0 2024-03-25T18:23:06.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.033+0000 7fad51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:06.045 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.044+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377228600 0 0) 0x7fad5c1d51f0 con 0x7fad5c0a98f0 2024-03-25T18:23:06.045 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.044+0000 7fad51ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:06.045 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.044+0000 7fad51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:06.045 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.044+0000 7fad51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:06.045 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.044+0000 7fad51ffb640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:23:06.045 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.044+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad40004eb0 con 0x7fad5c0a98f0 2024-03-25T18:23:06.046 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.045+0000 7fad51ffb640 10 client.?.objecter ms_handle_connect 0x7fad5c13f710 2024-03-25T18:23:06.046 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.045+0000 7fad51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:06.049 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.048+0000 7fad51ffb640 10 client.?.objecter ms_handle_connect 0x7fad5c1430c0 2024-03-25T18:23:06.049 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.048+0000 7fad51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:06.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.050+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660004249 0 0) 0x7fad5c1d7590 con 0x7fad5c13f710 2024-03-25T18:23:06.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.050+0000 7fad51ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:06.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.050+0000 7fad51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:06.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.050+0000 7fad51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:06.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.050+0000 7fad51ffb640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:23:06.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.050+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad5c1d51f0 con 0x7fad5c13f710 2024-03-25T18:23:06.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.059+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 982759036 0 0) 0x7fad40004eb0 con 0x7fad5c0a98f0 2024-03-25T18:23:06.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.059+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad5c1d7590 con 0x7fad5c0a98f0 2024-03-25T18:23:06.061 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.059+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fad58001630 con 0x7fad5c0a98f0 2024-03-25T18:23:06.061 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.059+0000 7fad51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:06.061 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.059+0000 7fad51ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:06.061 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.059+0000 7fad51ffb640 10 monclient(hunting): dump: 2024-03-25T18:23:06.061 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:06.061 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:06.061 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:06.061 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:06.061 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:06.061 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:06.061 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:06.061 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:06.061 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:06.061 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:06.061 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.059+0000 7fad51ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:06.063 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.062+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2508351764 0 0) 0x7fad5c1d51f0 con 0x7fad5c13f710 2024-03-25T18:23:06.063 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.062+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad40004eb0 con 0x7fad5c13f710 2024-03-25T18:23:06.063 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.062+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fad4c001930 con 0x7fad5c13f710 2024-03-25T18:23:06.063 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.062+0000 7fad51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:06.063 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.062+0000 7fad51ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:06.063 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.062+0000 7fad51ffb640 10 monclient(hunting): dump: 2024-03-25T18:23:06.063 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:06.063 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:06.063 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:06.064 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:06.064 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:06.064 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:06.064 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:06.064 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:06.064 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:06.064 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:06.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.067+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281687533 0 0) 0x7fad5c138fc0 con 0x7fad5c1430c0 2024-03-25T18:23:06.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.068+0000 7fad51ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:06.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.068+0000 7fad51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:06.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.068+0000 7fad51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:06.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.068+0000 7fad51ffb640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:23:06.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.068+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad5c1d51f0 con 0x7fad5c1430c0 2024-03-25T18:23:06.072 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.070+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 663744186 0 0) 0x7fad40004eb0 con 0x7fad5c13f710 2024-03-25T18:23:06.072 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 >> v1:172.21.15.122:6790/0 conn(0x7fad5c0a98f0 legacy=0x7fad5c1388b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 >> v1:172.21.15.122:6789/0 conn(0x7fad5c1430c0 legacy=0x7fad5c1e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad51ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad51ffb640 1 monclient: found mon.b 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad51ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad5c1d8460 con 0x7fad5c13f710 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad51ffb640 10 monclient: _finish_auth 0 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad51ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad6348d640 5 monclient: authenticate success, global_id 4522 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad6348d640 10 monclient: _renew_subs 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad6348d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad6348d640 1 -- 172.21.15.122:0/3332575313 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad5c0abe50 con 0x7fad5c13f710 2024-03-25T18:23:06.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad6348d640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:06.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad6348d640 10 monclient: _renew_subs 2024-03-25T18:23:06.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad6348d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:06.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad6348d640 1 -- 172.21.15.122:0/3332575313 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad5c1381d0 con 0x7fad5c13f710 2024-03-25T18:23:06.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.071+0000 7fad6348d640 1 librados: init done 2024-03-25T18:23:06.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.072+0000 7fad6348d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:06.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.072+0000 7fad6348d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:06.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.072+0000 7fad6348d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:06.075 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.072+0000 7fad6348d640 1 -- 172.21.15.122:0/3332575313 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad5c138470 con 0x7fad5c13f710 2024-03-25T18:23:06.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.097+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad4c001d40 con 0x7fad5c13f710 2024-03-25T18:23:06.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.097+0000 7fad51ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:06.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.097+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fad4c004980 con 0x7fad5c13f710 2024-03-25T18:23:06.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.097+0000 7fad51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:06.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.097+0000 7fad51ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:06.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.097+0000 7fad51ffb640 10 monclient: dump: 2024-03-25T18:23:06.099 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:06.099 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:06.099 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:06.099 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:06.100 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:06.100 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:06.100 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:06.100 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:06.100 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:06.100 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:06.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.097+0000 7fad537fe640 4 set_mon_vals no callback set 2024-03-25T18:23:06.101 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.098+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fad4c004ba0 con 0x7fad5c13f710 2024-03-25T18:23:06.101 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.099+0000 7fad61229640 10 monclient: get_auth_request con 0x7fad4006dee0 auth_method 0 2024-03-25T18:23:06.101 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.099+0000 7fad51ffb640 10 client.4522.objecter ms_handle_connect 0x7fad4006dee0 2024-03-25T18:23:06.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.101+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fad4c054740 con 0x7fad5c13f710 2024-03-25T18:23:06.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.101+0000 7fad51ffb640 10 client.4522.objecter ms_dispatch 0x7fad5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:06.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.101+0000 7fad51ffb640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:06.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.101+0000 7fad51ffb640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:06.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.101+0000 7fad51ffb640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-25T18:23:06.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.362+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fad4c050080 con 0x7fad5c13f710 2024-03-25T18:23:06.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.362+0000 7fad51ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:06.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.362+0000 7fad51ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:06.610 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.609+0000 7fad6348d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:06.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.609+0000 7fad6348d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:06.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.609+0000 7fad6348d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:06.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.609+0000 7fad6348d640 1 -- 172.21.15.122:0/3332575313 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fad5c0627b0 con 0x7fad5c13f710 2024-03-25T18:23:06.673 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:07:00.416 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:06.723 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:07:00.466 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:06.763 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:07:00.506 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:06.778 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:07:00.521 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:06.792 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:07:00.534 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:06.797 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:07:00.543 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:06.798 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.796+0000 7fad51ffb640 1 -- 172.21.15.122:0/3332575313 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fad4c052180 con 0x7fad5c13f710 2024-03-25T18:23:06.798 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.796+0000 7fad51ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:06.798 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.796+0000 7fad51ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:06.798 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:23:06.798 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:06.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.798+0000 7fad6348d640 10 librados: watch_flush enter 2024-03-25T18:23:06.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.798+0000 7fad6348d640 10 librados: watch_flush exit 2024-03-25T18:23:06.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.798+0000 7fad6348d640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-25T18:23:06.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.798+0000 7fad6348d640 10 client.4522.objecter successfully canceled tick 2024-03-25T18:23:06.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.799+0000 7fad6348d640 1 -- 172.21.15.122:0/3332575313 >> v1:172.21.15.204:6800/676228005 conn(0x7fad4006dee0 legacy=0x7fad400703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:06.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.799+0000 7fad6348d640 10 monclient: shutdown 2024-03-25T18:23:06.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.799+0000 7fad6348d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:06.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.799+0000 7fad6348d640 1 -- 172.21.15.122:0/3332575313 >> v1:172.21.15.204:6789/0 conn(0x7fad5c13f710 legacy=0x7fad5c1d7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:06.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.799+0000 7fad6348d640 1 -- 172.21.15.122:0/3332575313 shutdown_connections 2024-03-25T18:23:06.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.799+0000 7fad6348d640 1 -- 172.21.15.122:0/3332575313 >> 172.21.15.122:0/3332575313 conn(0x7fad5c0b7ea0 msgr2=0x7fad5c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:06.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.799+0000 7fad6348d640 1 -- 172.21.15.122:0/3332575313 shutdown_connections 2024-03-25T18:23:06.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.799+0000 7fad6348d640 1 -- 172.21.15.122:0/3332575313 wait complete. 2024-03-25T18:23:06.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:06.799+0000 7fad6348d640 1 librados: shutdown 2024-03-25T18:23:06.802 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:07:00.547 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:06.810 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:06.826 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:07:00.568 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:06.832 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:07:00.577 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:13.617 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:13.733 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.732+0000 7f536864e640 10 monclient: get_monmap_and_config 2024-03-25T18:23:13.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.732+0000 7f536864e640 10 monclient: build_initial_monmap 2024-03-25T18:23:13.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.732+0000 7f536864e640 10 monclient: monmap: 2024-03-25T18:23:13.734 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:23:13.734 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:13.734 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:23:13.733789+0000 2024-03-25T18:23:13.734 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:23:13.733789+0000 2024-03-25T18:23:13.734 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:13.734 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:13.734 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:23:13.735 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:23:13.735 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:23:13.735 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:13.736 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.733+0000 7f536864e640 1 Processor -- start 2024-03-25T18:23:13.736 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.733+0000 7f536864e640 1 -- start start 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.733+0000 7f536864e640 10 monclient: init 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.734+0000 7f536864e640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.734+0000 7f536864e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.734+0000 7f536864e640 10 monclient(hunting): picked mon.noname-b con 0x7f53600a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.734+0000 7f536864e640 10 monclient(hunting): picked mon.noname-a con 0x7f536013f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.734+0000 7f536864e640 10 monclient(hunting): picked mon.noname-c con 0x7f53601430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.734+0000 7f536864e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5360142520 con 0x7f536013f710 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.734+0000 7f536864e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53600b28a0 con 0x7f53601430c0 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.734+0000 7f536864e640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53600b2a30 con 0x7f53600a98f0 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.734+0000 7f536864e640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:13.737 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.734+0000 7f536864e640 10 monclient(hunting): authenticate will time out at 7717.000699s 2024-03-25T18:23:13.745 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.744+0000 7f53663ea640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f53600a98f0 0x7f53600a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:41454/0 (socket says 172.21.15.122:41454) 2024-03-25T18:23:13.746 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.744+0000 7f53663ea640 1 -- 172.21.15.122:0/2244602178 learned_addr learned my addr 172.21.15.122:0/2244602178 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:23:13.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.764+0000 7f53653e8640 1 -- 172.21.15.122:0/2244602178 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659703602 0 0) 0x7f53600b2a30 con 0x7f53600a98f0 2024-03-25T18:23:13.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.764+0000 7f53653e8640 10 monclient(hunting): _negotiate 2024-03-25T18:23:13.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.764+0000 7f53653e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:13.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.764+0000 7f53653e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:13.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.764+0000 7f53653e8640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:23:13.766 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.764+0000 7f53653e8640 1 -- 172.21.15.122:0/2244602178 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f534c003520 con 0x7f53600a98f0 2024-03-25T18:23:13.776 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.775+0000 7f53653e8640 1 -- 172.21.15.122:0/2244602178 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 76681219 0 0) 0x7f53600b28a0 con 0x7f53601430c0 2024-03-25T18:23:13.777 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.775+0000 7f53653e8640 10 monclient(hunting): _negotiate 2024-03-25T18:23:13.777 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.775+0000 7f53653e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:13.777 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.775+0000 7f53653e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:13.777 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.775+0000 7f53653e8640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:23:13.777 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.775+0000 7f53653e8640 1 -- 172.21.15.122:0/2244602178 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53600b2a30 con 0x7f53601430c0 2024-03-25T18:23:13.784 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.783+0000 7f53653e8640 1 -- 172.21.15.122:0/2244602178 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48784295 0 0) 0x7f5360142520 con 0x7f536013f710 2024-03-25T18:23:13.784 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.783+0000 7f53653e8640 10 monclient(hunting): _negotiate 2024-03-25T18:23:13.784 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.783+0000 7f53653e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:13.784 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.783+0000 7f53653e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:13.784 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.783+0000 7f53653e8640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:23:13.784 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.783+0000 7f53653e8640 1 -- 172.21.15.122:0/2244602178 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53600b28a0 con 0x7f536013f710 2024-03-25T18:23:13.786 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.784+0000 7f53653e8640 1 -- 172.21.15.122:0/2244602178 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 604300783 0 0) 0x7f53600b2a30 con 0x7f53601430c0 2024-03-25T18:23:13.786 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.784+0000 7f53653e8640 1 -- 172.21.15.122:0/2244602178 >> v1:172.21.15.122:6790/0 conn(0x7f53600a98f0 legacy=0x7f53600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:13.786 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.784+0000 7f53653e8640 1 -- 172.21.15.122:0/2244602178 >> v1:172.21.15.122:6789/0 conn(0x7f536013f710 legacy=0x7f53601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:13.786 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.784+0000 7f53653e8640 10 monclient: _finish_hunting 0 2024-03-25T18:23:13.786 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.784+0000 7f53653e8640 1 monclient: found mon.noname-c 2024-03-25T18:23:13.786 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.784+0000 7f53653e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:13.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.784+0000 7f53653e8640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:23:13.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.784+0000 7f53653e8640 1 -- 172.21.15.122:0/2244602178 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53600b44c0 con 0x7f53601430c0 2024-03-25T18:23:13.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.784+0000 7f53653e8640 10 monclient: _finish_auth 0 2024-03-25T18:23:13.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.784+0000 7f53653e8640 10 monclient: _check_auth_tickets 2024-03-25T18:23:13.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.784+0000 7f53653e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:13.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.784+0000 7f53653e8640 1 -- 172.21.15.122:0/2244602178 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f53580019f0 con 0x7f53601430c0 2024-03-25T18:23:13.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.784+0000 7f53653e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:13.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.784+0000 7f53653e8640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:13.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.784+0000 7f53653e8640 10 monclient: dump: 2024-03-25T18:23:13.787 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:13.787 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:13.787 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:13.787 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:13.787 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:13.788 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:13.788 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:13.788 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:13.788 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:13.788 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:13.788 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.784+0000 7f536864e640 5 monclient: authenticate success, global_id 4525 2024-03-25T18:23:13.788 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.784+0000 7f536864e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:13.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.788+0000 7f53653e8640 1 -- 172.21.15.122:0/2244602178 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5358001b50 con 0x7f53601430c0 2024-03-25T18:23:13.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.788+0000 7f53653e8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:13.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.788+0000 7f536864e640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:13.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.788+0000 7f536864e640 4 set_mon_vals no callback set 2024-03-25T18:23:13.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.789+0000 7f536864e640 10 monclient: shutdown 2024-03-25T18:23:13.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.789+0000 7f536864e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:13.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.789+0000 7f536864e640 1 -- 172.21.15.122:0/2244602178 >> v1:172.21.15.204:6789/0 conn(0x7f53601430c0 legacy=0x7f536014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:13.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.789+0000 7f536864e640 1 -- 172.21.15.122:0/2244602178 shutdown_connections 2024-03-25T18:23:13.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.789+0000 7f536864e640 1 -- 172.21.15.122:0/2244602178 >> 172.21.15.122:0/2244602178 conn(0x7f53600b7ea0 msgr2=0x7f53600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:13.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.789+0000 7f536864e640 1 -- 172.21.15.122:0/2244602178 shutdown_connections 2024-03-25T18:23:13.791 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.789+0000 7f536864e640 1 -- 172.21.15.122:0/2244602178 wait complete. 2024-03-25T18:23:13.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.790+0000 7f536864e640 10 monclient: build_initial_monmap 2024-03-25T18:23:13.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.790+0000 7f536864e640 10 monclient: monmap: 2024-03-25T18:23:13.793 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:23:13.793 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:13.793 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:23:13.793 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:23:13.793 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:13.793 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:23:13.793 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:23:13.793 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:23:13.793 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:23:13.794 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:13.794 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.790+0000 7f536864e640 1 librados: starting msgr at 2024-03-25T18:23:13.794 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.790+0000 7f536864e640 1 librados: starting objecter 2024-03-25T18:23:13.794 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.790+0000 7f536864e640 1 Processor -- start 2024-03-25T18:23:13.794 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.791+0000 7f536864e640 1 -- start start 2024-03-25T18:23:13.794 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.791+0000 7f536864e640 1 librados: setting wanted keys 2024-03-25T18:23:13.794 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.791+0000 7f536864e640 1 librados: calling monclient init 2024-03-25T18:23:13.794 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.791+0000 7f536864e640 10 monclient: init 2024-03-25T18:23:13.794 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.791+0000 7f536864e640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:13.794 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.791+0000 7f536864e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:13.794 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.791+0000 7f536864e640 10 monclient(hunting): picked mon.noname-c con 0x7f53600a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:23:13.794 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.791+0000 7f536864e640 10 monclient(hunting): picked mon.noname-a con 0x7f536013f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:23:13.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.791+0000 7f536864e640 10 monclient(hunting): picked mon.noname-b con 0x7f53601430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:23:13.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.791+0000 7f536864e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5360138400 con 0x7f536013f710 2024-03-25T18:23:13.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.791+0000 7f536864e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5360138fa0 con 0x7f53601430c0 2024-03-25T18:23:13.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.791+0000 7f536864e640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53601d51f0 con 0x7f53600a98f0 2024-03-25T18:23:13.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.791+0000 7f536864e640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:13.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.791+0000 7f536864e640 10 monclient(hunting): authenticate will time out at 7717.057845s 2024-03-25T18:23:13.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.803+0000 7f53663ea640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f53600a98f0 0x7f5360138890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:41508/0 (socket says 172.21.15.122:41508) 2024-03-25T18:23:13.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.803+0000 7f53663ea640 1 -- 172.21.15.122:0/2998215061 learned_addr learned my addr 172.21.15.122:0/2998215061 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:23:13.807 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.806+0000 7f5356ffd640 10 client.?.objecter ms_handle_connect 0x7f53600a98f0 2024-03-25T18:23:13.807 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.806+0000 7f5356ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:13.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.808+0000 7f5356ffd640 10 client.?.objecter ms_handle_connect 0x7f536013f710 2024-03-25T18:23:13.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.808+0000 7f5356ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:13.814 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.813+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2807548769 0 0) 0x7f5360138400 con 0x7f536013f710 2024-03-25T18:23:13.815 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.813+0000 7f5356ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:13.815 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.813+0000 7f5356ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:13.815 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.813+0000 7f5356ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:13.815 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.813+0000 7f5356ffd640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:23:13.815 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.813+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5344004eb0 con 0x7f536013f710 2024-03-25T18:23:13.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.819+0000 7f5356ffd640 10 client.?.objecter ms_handle_connect 0x7f53601430c0 2024-03-25T18:23:13.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.819+0000 7f5356ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:13.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.819+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1233040793 0 0) 0x7f53601d51f0 con 0x7f53600a98f0 2024-03-25T18:23:13.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.819+0000 7f5356ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:13.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.819+0000 7f5356ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:13.821 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.819+0000 7f5356ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:13.821 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.819+0000 7f5356ffd640 10 monclient(hunting): my global_id is 4499 2024-03-25T18:23:13.821 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.819+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5360138400 con 0x7f53600a98f0 2024-03-25T18:23:13.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.822+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3491008201 0 0) 0x7f5344004eb0 con 0x7f536013f710 2024-03-25T18:23:13.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.822+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53601d51f0 con 0x7f536013f710 2024-03-25T18:23:13.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.822+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f53500018f0 con 0x7f536013f710 2024-03-25T18:23:13.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.822+0000 7f5356ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:13.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.822+0000 7f5356ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:13.824 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.822+0000 7f5356ffd640 10 monclient(hunting): dump: 2024-03-25T18:23:13.824 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:13.824 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:13.824 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:13.824 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:13.824 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:13.824 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:13.824 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:13.824 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:13.824 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:13.824 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:13.824 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.822+0000 7f5356ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:13.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.825+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3338877305 0 0) 0x7f5360138fa0 con 0x7f53601430c0 2024-03-25T18:23:13.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.825+0000 7f5356ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:13.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.825+0000 7f5356ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:13.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.825+0000 7f5356ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:13.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.825+0000 7f5356ffd640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:23:13.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.825+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5344004eb0 con 0x7f53601430c0 2024-03-25T18:23:13.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.832+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3700595548 0 0) 0x7f5360138400 con 0x7f53600a98f0 2024-03-25T18:23:13.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.832+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5360138fa0 con 0x7f53600a98f0 2024-03-25T18:23:13.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.832+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f535c0018d0 con 0x7f53600a98f0 2024-03-25T18:23:13.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.832+0000 7f5356ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:13.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.832+0000 7f5356ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:13.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.832+0000 7f5356ffd640 10 monclient(hunting): dump: 2024-03-25T18:23:13.834 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:13.834 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:13.834 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:13.834 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:13.834 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:13.834 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:13.834 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:13.834 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:13.834 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:13.834 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:13.837 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2372630942 0 0) 0x7f53601d51f0 con 0x7f536013f710 2024-03-25T18:23:13.838 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 >> v1:172.21.15.122:6790/0 conn(0x7f53600a98f0 legacy=0x7f5360138890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:13.838 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 >> v1:172.21.15.204:6789/0 conn(0x7f53601430c0 legacy=0x7f53601db4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:13.838 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f5356ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:13.838 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f5356ffd640 1 monclient: found mon.a 2024-03-25T18:23:13.838 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f5356ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:13.838 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f5356ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:23:13.838 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5360138020 con 0x7f536013f710 2024-03-25T18:23:13.838 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f5356ffd640 10 monclient: _finish_auth 0 2024-03-25T18:23:13.838 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f5356ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:13.838 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f5356ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:13.838 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f536864e640 5 monclient: authenticate success, global_id 4536 2024-03-25T18:23:13.838 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f536864e640 10 monclient: _renew_subs 2024-03-25T18:23:13.839 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f536864e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:23:13.839 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f536864e640 1 -- 172.21.15.122:0/2998215061 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53601e2ed0 con 0x7f536013f710 2024-03-25T18:23:13.839 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f536864e640 10 client.4536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:13.839 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f536864e640 10 monclient: _renew_subs 2024-03-25T18:23:13.839 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f536864e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:23:13.839 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f536864e640 1 -- 172.21.15.122:0/2998215061 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53601e3240 con 0x7f536013f710 2024-03-25T18:23:13.839 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.836+0000 7f536864e640 1 librados: init done 2024-03-25T18:23:13.839 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.837+0000 7f536864e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:13.839 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.837+0000 7f536864e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:13.840 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.837+0000 7f536864e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:23:13.840 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:13.837+0000 7f536864e640 1 -- 172.21.15.122:0/2998215061 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53600627b0 con 0x7f536013f710 2024-03-25T18:23:14.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.140+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5350003610 con 0x7f536013f710 2024-03-25T18:23:14.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.140+0000 7f5356ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:14.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.140+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5350003920 con 0x7f536013f710 2024-03-25T18:23:14.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.140+0000 7f5356ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:14.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.140+0000 7f5356ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:14.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.140+0000 7f5356ffd640 10 monclient: dump: 2024-03-25T18:23:14.142 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:14.142 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:14.142 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:14.142 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:14.143 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:14.143 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:14.143 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:14.143 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:14.143 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:14.143 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:14.143 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.140+0000 7f5364be7640 4 set_mon_vals no callback set 2024-03-25T18:23:14.144 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.141+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f53500198f0 con 0x7f536013f710 2024-03-25T18:23:14.144 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.142+0000 7f53663ea640 10 monclient: get_auth_request con 0x7f534406dfb0 auth_method 0 2024-03-25T18:23:14.144 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.142+0000 7f5356ffd640 10 client.4536.objecter ms_handle_connect 0x7f534406dfb0 2024-03-25T18:23:14.148 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.147+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f5350053550 con 0x7f536013f710 2024-03-25T18:23:14.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.147+0000 7f5356ffd640 10 client.4536.objecter ms_dispatch 0x7f5360000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:14.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.147+0000 7f5356ffd640 3 client.4536.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:14.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.147+0000 7f5356ffd640 3 client.4536.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:14.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.147+0000 7f5356ffd640 20 client.4536.objecter dump_active .. 0 homeless 2024-03-25T18:23:14.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.147+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f535004f060 con 0x7f536013f710 2024-03-25T18:23:14.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.147+0000 7f5356ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:14.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.147+0000 7f5356ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:14.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.405+0000 7f536864e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:14.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.405+0000 7f536864e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:14.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.405+0000 7f536864e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:23:14.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.405+0000 7f536864e640 1 -- 172.21.15.122:0/2998215061 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53601385b0 con 0x7f536013f710 2024-03-25T18:23:14.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.434+0000 7f5356ffd640 1 -- 172.21.15.122:0/2998215061 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f53601385b0 con 0x7f536013f710 2024-03-25T18:23:14.436 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.434+0000 7f5356ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:14.436 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.434+0000 7f5356ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:14.436 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:23:14.436 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:14.439 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.437+0000 7f536864e640 10 librados: watch_flush enter 2024-03-25T18:23:14.439 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.437+0000 7f536864e640 10 librados: watch_flush exit 2024-03-25T18:23:14.439 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.437+0000 7f536864e640 20 client.4536.objecter shutdown clearing up homeless session... 2024-03-25T18:23:14.439 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.437+0000 7f536864e640 10 client.4536.objecter successfully canceled tick 2024-03-25T18:23:14.439 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.437+0000 7f536864e640 1 -- 172.21.15.122:0/2998215061 >> v1:172.21.15.204:6800/676228005 conn(0x7f534406dfb0 legacy=0x7f5344070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:14.439 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.437+0000 7f536864e640 10 monclient: shutdown 2024-03-25T18:23:14.439 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.437+0000 7f536864e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:14.439 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.437+0000 7f536864e640 1 -- 172.21.15.122:0/2998215061 >> v1:172.21.15.122:6789/0 conn(0x7f536013f710 legacy=0x7f53601dade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:14.439 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.437+0000 7f536864e640 1 -- 172.21.15.122:0/2998215061 shutdown_connections 2024-03-25T18:23:14.439 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.437+0000 7f536864e640 1 -- 172.21.15.122:0/2998215061 >> 172.21.15.122:0/2998215061 conn(0x7f53600b7ea0 msgr2=0x7f5360074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:14.439 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.437+0000 7f536864e640 1 -- 172.21.15.122:0/2998215061 shutdown_connections 2024-03-25T18:23:14.440 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.437+0000 7f536864e640 1 -- 172.21.15.122:0/2998215061 wait complete. 2024-03-25T18:23:14.440 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:14.437+0000 7f536864e640 1 librados: shutdown 2024-03-25T18:23:14.450 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:21.256 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:21.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.371+0000 7fcfafc4e640 10 monclient: get_monmap_and_config 2024-03-25T18:23:21.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.371+0000 7fcfafc4e640 10 monclient: build_initial_monmap 2024-03-25T18:23:21.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.371+0000 7fcfafc4e640 10 monclient: monmap: 2024-03-25T18:23:21.372 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:23:21.372 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:21.372 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:23:21.372204+0000 2024-03-25T18:23:21.372 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:23:21.372204+0000 2024-03-25T18:23:21.372 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:21.372 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:21.372 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:23:21.372 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:23:21.373 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:23:21.373 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:21.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.371+0000 7fcfafc4e640 1 Processor -- start 2024-03-25T18:23:21.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.371+0000 7fcfafc4e640 1 -- start start 2024-03-25T18:23:21.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.371+0000 7fcfafc4e640 10 monclient: init 2024-03-25T18:23:21.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.372+0000 7fcfafc4e640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:21.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.372+0000 7fcfafc4e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:21.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.372+0000 7fcfafc4e640 10 monclient(hunting): picked mon.noname-a con 0x7fcfa80b4d60 addr v1:172.21.15.122:6789/0 2024-03-25T18:23:21.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.372+0000 7fcfafc4e640 10 monclient(hunting): picked mon.noname-c con 0x7fcfa80b1730 addr v1:172.21.15.204:6789/0 2024-03-25T18:23:21.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.372+0000 7fcfafc4e640 10 monclient(hunting): picked mon.noname-b con 0x7fcfa814be90 addr v1:172.21.15.122:6790/0 2024-03-25T18:23:21.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.372+0000 7fcfafc4e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa80af940 con 0x7fcfa80b4d60 2024-03-25T18:23:21.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.372+0000 7fcfafc4e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa80b0a70 con 0x7fcfa80b1730 2024-03-25T18:23:21.375 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.372+0000 7fcfafc4e640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa80b0d50 con 0x7fcfa814be90 2024-03-25T18:23:21.375 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.372+0000 7fcfafc4e640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:21.375 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.372+0000 7fcfafc4e640 10 monclient(hunting): authenticate will time out at 7724.638561s 2024-03-25T18:23:21.382 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.381+0000 7fcfad9ea640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fcfa80b4d60 0x7fcfa813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:52516/0 (socket says 172.21.15.122:52516) 2024-03-25T18:23:21.382 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.381+0000 7fcfad9ea640 1 -- 172.21.15.122:0/2632108318 learned_addr learned my addr 172.21.15.122:0/2632108318 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:23:21.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.393+0000 7fcfac9e8640 1 -- 172.21.15.122:0/2632108318 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901010021 0 0) 0x7fcfa80af940 con 0x7fcfa80b4d60 2024-03-25T18:23:21.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.393+0000 7fcfac9e8640 10 monclient(hunting): _negotiate 2024-03-25T18:23:21.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.393+0000 7fcfac9e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:21.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.393+0000 7fcfac9e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:21.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.393+0000 7fcfac9e8640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:23:21.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.393+0000 7fcfac9e8640 1 -- 172.21.15.122:0/2632108318 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf94003520 con 0x7fcfa80b4d60 2024-03-25T18:23:21.401 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.400+0000 7fcfac9e8640 1 -- 172.21.15.122:0/2632108318 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2604560800 0 0) 0x7fcfa80b0d50 con 0x7fcfa814be90 2024-03-25T18:23:21.401 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.400+0000 7fcfac9e8640 10 monclient(hunting): _negotiate 2024-03-25T18:23:21.401 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.400+0000 7fcfac9e8640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:21.401 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.400+0000 7fcfac9e8640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:21.401 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.400+0000 7fcfac9e8640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:23:21.401 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.400+0000 7fcfac9e8640 1 -- 172.21.15.122:0/2632108318 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfa80af940 con 0x7fcfa814be90 2024-03-25T18:23:21.405 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfac9e8640 1 -- 172.21.15.122:0/2632108318 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 745579177 0 0) 0x7fcf94003520 con 0x7fcfa80b4d60 2024-03-25T18:23:21.405 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfac9e8640 1 -- 172.21.15.122:0/2632108318 >> v1:172.21.15.122:6790/0 conn(0x7fcfa814be90 legacy=0x7fcfa814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:21.405 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfac9e8640 1 -- 172.21.15.122:0/2632108318 >> v1:172.21.15.204:6789/0 conn(0x7fcfa80b1730 legacy=0x7fcfa80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:21.405 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfac9e8640 10 monclient: _finish_hunting 0 2024-03-25T18:23:21.405 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfac9e8640 1 monclient: found mon.noname-a 2024-03-25T18:23:21.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfac9e8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:21.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfac9e8640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:23:21.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfac9e8640 1 -- 172.21.15.122:0/2632108318 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa80b1030 con 0x7fcfa80b4d60 2024-03-25T18:23:21.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfac9e8640 10 monclient: _finish_auth 0 2024-03-25T18:23:21.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfac9e8640 10 monclient: _check_auth_tickets 2024-03-25T18:23:21.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfac9e8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:21.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfac9e8640 1 -- 172.21.15.122:0/2632108318 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fcfa4001690 con 0x7fcfa80b4d60 2024-03-25T18:23:21.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfafc4e640 5 monclient: authenticate success, global_id 4539 2024-03-25T18:23:21.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfafc4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:21.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfac9e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:21.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfac9e8640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:21.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfac9e8640 10 monclient: dump: 2024-03-25T18:23:21.406 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:21.407 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:21.407 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:21.407 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:21.407 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:21.407 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:21.407 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:21.407 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:21.407 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:21.407 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:21.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.404+0000 7fcfafc4e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:21.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.413+0000 7fcfac9e8640 1 -- 172.21.15.122:0/2632108318 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfa40017f0 con 0x7fcfa80b4d60 2024-03-25T18:23:21.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.413+0000 7fcfac9e8640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:21.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.413+0000 7fcfac9e8640 1 -- 172.21.15.122:0/2632108318 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fcfa4001c00 con 0x7fcfa80b4d60 2024-03-25T18:23:21.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.413+0000 7fcfac9e8640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:21.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.413+0000 7fcfac9e8640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:21.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.413+0000 7fcfac9e8640 10 monclient: dump: 2024-03-25T18:23:21.415 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:21.415 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:21.415 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:21.415 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:21.415 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:21.415 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:21.415 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:21.415 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:21.415 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:21.415 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:21.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.413+0000 7fcfafc4e640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:21.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.413+0000 7fcfafc4e640 4 set_mon_vals no callback set 2024-03-25T18:23:21.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.414+0000 7fcfafc4e640 10 monclient: shutdown 2024-03-25T18:23:21.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.414+0000 7fcfafc4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:21.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.414+0000 7fcfafc4e640 1 -- 172.21.15.122:0/2632108318 >> v1:172.21.15.122:6789/0 conn(0x7fcfa80b4d60 legacy=0x7fcfa813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:21.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.414+0000 7fcfafc4e640 1 -- 172.21.15.122:0/2632108318 shutdown_connections 2024-03-25T18:23:21.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.414+0000 7fcfafc4e640 1 -- 172.21.15.122:0/2632108318 >> 172.21.15.122:0/2632108318 conn(0x7fcfa80617a0 msgr2=0x7fcfa8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:21.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.414+0000 7fcfafc4e640 1 -- 172.21.15.122:0/2632108318 shutdown_connections 2024-03-25T18:23:21.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.414+0000 7fcfafc4e640 1 -- 172.21.15.122:0/2632108318 wait complete. 2024-03-25T18:23:21.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.415+0000 7fcfafc4e640 10 monclient: build_initial_monmap 2024-03-25T18:23:21.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.415+0000 7fcfafc4e640 10 monclient: monmap: 2024-03-25T18:23:21.418 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:23:21.418 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:21.418 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:23:21.419 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:23:21.419 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:21.419 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:23:21.419 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:23:21.419 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:23:21.419 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:23:21.419 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:21.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.415+0000 7fcfafc4e640 1 librados: starting msgr at 2024-03-25T18:23:21.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.415+0000 7fcfafc4e640 1 librados: starting objecter 2024-03-25T18:23:21.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.415+0000 7fcfafc4e640 1 Processor -- start 2024-03-25T18:23:21.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.415+0000 7fcfafc4e640 1 -- start start 2024-03-25T18:23:21.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.415+0000 7fcfafc4e640 1 librados: setting wanted keys 2024-03-25T18:23:21.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.415+0000 7fcfafc4e640 1 librados: calling monclient init 2024-03-25T18:23:21.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.415+0000 7fcfafc4e640 10 monclient: init 2024-03-25T18:23:21.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.416+0000 7fcfafc4e640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:21.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.416+0000 7fcfafc4e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:21.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.416+0000 7fcfafc4e640 10 monclient(hunting): picked mon.noname-c con 0x7fcfa80b1730 addr v1:172.21.15.122:6790/0 2024-03-25T18:23:21.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.416+0000 7fcfafc4e640 10 monclient(hunting): picked mon.noname-b con 0x7fcfa80b4d60 addr v1:172.21.15.204:6789/0 2024-03-25T18:23:21.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.416+0000 7fcfafc4e640 10 monclient(hunting): picked mon.noname-a con 0x7fcfa814be90 addr v1:172.21.15.122:6789/0 2024-03-25T18:23:21.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.416+0000 7fcfafc4e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa8138210 con 0x7fcfa814be90 2024-03-25T18:23:21.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.416+0000 7fcfafc4e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa81375a0 con 0x7fcfa80b4d60 2024-03-25T18:23:21.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.416+0000 7fcfafc4e640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfa8138440 con 0x7fcfa80b1730 2024-03-25T18:23:21.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.416+0000 7fcfafc4e640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:21.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.416+0000 7fcfafc4e640 10 monclient(hunting): authenticate will time out at 7724.682696s 2024-03-25T18:23:21.424 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.423+0000 7fcfae1eb640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fcfa814be90 0x7fcfa81e3280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:52536/0 (socket says 172.21.15.122:52536) 2024-03-25T18:23:21.424 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.423+0000 7fcfae1eb640 1 -- 172.21.15.122:0/350912430 learned_addr learned my addr 172.21.15.122:0/350912430 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:23:21.424 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.423+0000 7fcf9e7fc640 10 client.?.objecter ms_handle_connect 0x7fcfa814be90 2024-03-25T18:23:21.425 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.423+0000 7fcf9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:21.427 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.426+0000 7fcf9e7fc640 10 client.?.objecter ms_handle_connect 0x7fcfa80b1730 2024-03-25T18:23:21.427 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.426+0000 7fcf9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:21.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.429+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502188429 0 0) 0x7fcfa8138210 con 0x7fcfa814be90 2024-03-25T18:23:21.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.429+0000 7fcf9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:21.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.429+0000 7fcf9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:21.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.429+0000 7fcf9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:21.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.429+0000 7fcf9e7fc640 10 monclient(hunting): my global_id is 4542 2024-03-25T18:23:21.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.429+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf8c004eb0 con 0x7fcfa814be90 2024-03-25T18:23:21.431 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.430+0000 7fcf9e7fc640 10 client.?.objecter ms_handle_connect 0x7fcfa80b4d60 2024-03-25T18:23:21.431 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.430+0000 7fcf9e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:21.438 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.437+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4270394809 0 0) 0x7fcf8c004eb0 con 0x7fcfa814be90 2024-03-25T18:23:21.438 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.437+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfa8138210 con 0x7fcfa814be90 2024-03-25T18:23:21.439 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.437+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fcfa0001c90 con 0x7fcfa814be90 2024-03-25T18:23:21.439 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.437+0000 7fcf9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:21.439 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.437+0000 7fcf9e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:21.439 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.437+0000 7fcf9e7fc640 10 monclient(hunting): dump: 2024-03-25T18:23:21.439 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:21.439 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:21.439 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:21.439 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:21.439 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:21.439 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:21.439 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:21.439 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:21.439 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:21.439 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:21.440 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.437+0000 7fcf9e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:21.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.444+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151578008 0 0) 0x7fcfa8138440 con 0x7fcfa80b1730 2024-03-25T18:23:21.446 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.444+0000 7fcf9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:21.446 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.444+0000 7fcf9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:21.446 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.444+0000 7fcf9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:21.446 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.444+0000 7fcf9e7fc640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:23:21.446 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.444+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf8c004eb0 con 0x7fcfa80b1730 2024-03-25T18:23:21.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.448+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3023966470 0 0) 0x7fcfa81375a0 con 0x7fcfa80b4d60 2024-03-25T18:23:21.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.448+0000 7fcf9e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:21.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.448+0000 7fcf9e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:21.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.448+0000 7fcf9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:21.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.448+0000 7fcf9e7fc640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:23:21.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.448+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfa8138440 con 0x7fcfa80b4d60 2024-03-25T18:23:21.452 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1424857205 0 0) 0x7fcfa8138210 con 0x7fcfa814be90 2024-03-25T18:23:21.452 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 >> v1:172.21.15.122:6790/0 conn(0x7fcfa80b1730 legacy=0x7fcfa8139320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:21.452 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 >> v1:172.21.15.204:6789/0 conn(0x7fcfa80b4d60 legacy=0x7fcfa812fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:21.452 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcf9e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:21.452 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcf9e7fc640 1 monclient: found mon.a 2024-03-25T18:23:21.452 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcf9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:21.453 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcf9e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:23:21.453 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa81364b0 con 0x7fcfa814be90 2024-03-25T18:23:21.453 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcf9e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:21.453 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcf9e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:21.453 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcf9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:21.453 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcfafc4e640 5 monclient: authenticate success, global_id 4542 2024-03-25T18:23:21.453 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcfafc4e640 10 monclient: _renew_subs 2024-03-25T18:23:21.453 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcfafc4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:23:21.453 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcfafc4e640 1 -- 172.21.15.122:0/350912430 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa813b470 con 0x7fcfa814be90 2024-03-25T18:23:21.453 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcfafc4e640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:21.453 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcfafc4e640 10 monclient: _renew_subs 2024-03-25T18:23:21.453 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcfafc4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:23:21.453 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcfafc4e640 1 -- 172.21.15.122:0/350912430 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa8136230 con 0x7fcfa814be90 2024-03-25T18:23:21.453 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.450+0000 7fcfafc4e640 1 librados: init done 2024-03-25T18:23:21.454 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.451+0000 7fcfafc4e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:21.454 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.451+0000 7fcfafc4e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:21.454 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.451+0000 7fcfafc4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:23:21.454 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.451+0000 7fcfafc4e640 1 -- 172.21.15.122:0/350912430 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfa80627b0 con 0x7fcfa814be90 2024-03-25T18:23:21.763 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.761+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfa0003990 con 0x7fcfa814be90 2024-03-25T18:23:21.763 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.761+0000 7fcf9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:21.763 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.761+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fcfa0003da0 con 0x7fcfa814be90 2024-03-25T18:23:21.763 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.762+0000 7fcf9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:21.763 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.762+0000 7fcf9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:21.763 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.762+0000 7fcf9e7fc640 10 monclient: dump: 2024-03-25T18:23:21.764 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:21.764 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:21.764 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:21.764 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:21.764 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:21.764 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:21.764 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:21.764 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:21.764 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:21.764 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:21.764 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.761+0000 7fcf9ffff640 4 set_mon_vals no callback set 2024-03-25T18:23:21.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.762+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fcfa0003fc0 con 0x7fcfa814be90 2024-03-25T18:23:21.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.763+0000 7fcfad9ea640 10 monclient: get_auth_request con 0x7fcf8c06de80 auth_method 0 2024-03-25T18:23:21.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.763+0000 7fcf9e7fc640 10 client.4542.objecter ms_handle_connect 0x7fcf8c06de80 2024-03-25T18:23:21.782 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.781+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fcfa0054a50 con 0x7fcfa814be90 2024-03-25T18:23:21.782 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.781+0000 7fcf9e7fc640 10 client.4542.objecter ms_dispatch 0x7fcfa8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:21.782 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.781+0000 7fcf9e7fc640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:21.782 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.781+0000 7fcf9e7fc640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:21.782 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.781+0000 7fcf9e7fc640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-25T18:23:21.782 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.781+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcfa0050560 con 0x7fcfa814be90 2024-03-25T18:23:21.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.781+0000 7fcf9e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:21.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:21.781+0000 7fcf9e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:22.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.035+0000 7fcfafc4e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:22.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.035+0000 7fcfafc4e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:22.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.035+0000 7fcfafc4e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:23:22.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.035+0000 7fcfafc4e640 1 -- 172.21.15.122:0/350912430 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcfa8136640 con 0x7fcfa814be90 2024-03-25T18:23:22.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.080+0000 7fcf9e7fc640 1 -- 172.21.15.122:0/350912430 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fcfa0052490 con 0x7fcfa814be90 2024-03-25T18:23:22.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.080+0000 7fcf9e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:22.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.080+0000 7fcf9e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:22.082 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:23:22.083 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:22.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.083+0000 7fcfafc4e640 10 librados: watch_flush enter 2024-03-25T18:23:22.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.083+0000 7fcfafc4e640 10 librados: watch_flush exit 2024-03-25T18:23:22.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.083+0000 7fcfafc4e640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-25T18:23:22.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.083+0000 7fcfafc4e640 10 client.4542.objecter successfully canceled tick 2024-03-25T18:23:22.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.083+0000 7fcfafc4e640 1 -- 172.21.15.122:0/350912430 >> v1:172.21.15.204:6800/676228005 conn(0x7fcf8c06de80 legacy=0x7fcf8c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:22.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.083+0000 7fcfafc4e640 10 monclient: shutdown 2024-03-25T18:23:22.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.083+0000 7fcfafc4e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:22.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.083+0000 7fcfafc4e640 1 -- 172.21.15.122:0/350912430 >> v1:172.21.15.122:6789/0 conn(0x7fcfa814be90 legacy=0x7fcfa81e3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:22.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.083+0000 7fcfafc4e640 1 -- 172.21.15.122:0/350912430 shutdown_connections 2024-03-25T18:23:22.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.083+0000 7fcfafc4e640 1 -- 172.21.15.122:0/350912430 >> 172.21.15.122:0/350912430 conn(0x7fcfa80617a0 msgr2=0x7fcfa813f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:22.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.083+0000 7fcfafc4e640 1 -- 172.21.15.122:0/350912430 shutdown_connections 2024-03-25T18:23:22.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.083+0000 7fcfafc4e640 1 -- 172.21.15.122:0/350912430 wait complete. 2024-03-25T18:23:22.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:22.083+0000 7fcfafc4e640 1 librados: shutdown 2024-03-25T18:23:22.097 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:27.127 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:07:20.869 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:27.131 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:07:20.873 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:27.168 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:07:20.910 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:27.183 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:07:20.929 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:27.186 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:07:20.932 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:27.193 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:07:20.935 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:27.205 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:07:20.948 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:27.224 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:07:20.970 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:27.236 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:07:20.979 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:28.904 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:29.001 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.000+0000 7f467d9fb640 10 monclient: get_monmap_and_config 2024-03-25T18:23:29.001 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.000+0000 7f467d9fb640 10 monclient: build_initial_monmap 2024-03-25T18:23:29.001 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.000+0000 7f467d9fb640 10 monclient: monmap: 2024-03-25T18:23:29.001 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:23:29.001 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:29.001 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:23:29.001218+0000 2024-03-25T18:23:29.001 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:23:29.001218+0000 2024-03-25T18:23:29.001 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:29.001 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:29.001 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:23:29.002 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:23:29.002 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:23:29.002 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:29.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.000+0000 7f467d9fb640 1 Processor -- start 2024-03-25T18:23:29.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.001+0000 7f467d9fb640 1 -- start start 2024-03-25T18:23:29.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.001+0000 7f467d9fb640 10 monclient: init 2024-03-25T18:23:29.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.001+0000 7f467d9fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:29.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.001+0000 7f467d9fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:29.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.001+0000 7f467d9fb640 10 monclient(hunting): picked mon.noname-b con 0x7f46780a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:23:29.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.001+0000 7f467d9fb640 10 monclient(hunting): picked mon.noname-c con 0x7f467813f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:23:29.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.001+0000 7f467d9fb640 10 monclient(hunting): picked mon.noname-a con 0x7f46781430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:23:29.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.001+0000 7f467d9fb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4678142520 con 0x7f46781430c0 2024-03-25T18:23:29.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.002+0000 7f467d9fb640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46780b28a0 con 0x7f467813f710 2024-03-25T18:23:29.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.002+0000 7f467d9fb640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46780b2a30 con 0x7f46780a98f0 2024-03-25T18:23:29.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.002+0000 7f467d9fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:29.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.002+0000 7f467d9fb640 10 monclient(hunting): authenticate will time out at 7732.268020s 2024-03-25T18:23:29.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.004+0000 7f466ffff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f467813f710 0x7f46781429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:57850/0 (socket says 172.21.15.122:57850) 2024-03-25T18:23:29.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.004+0000 7f466ffff640 1 -- 172.21.15.122:0/2307620657 learned_addr learned my addr 172.21.15.122:0/2307620657 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:23:29.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.011+0000 7f46767fc640 1 -- 172.21.15.122:0/2307620657 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4023167899 0 0) 0x7f46780b28a0 con 0x7f467813f710 2024-03-25T18:23:29.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.011+0000 7f46767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:29.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.011+0000 7f46767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:29.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.011+0000 7f46767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:29.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.011+0000 7f46767fc640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:23:29.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.011+0000 7f46767fc640 1 -- 172.21.15.122:0/2307620657 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4664003520 con 0x7f467813f710 2024-03-25T18:23:29.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.019+0000 7f46767fc640 1 -- 172.21.15.122:0/2307620657 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1501251511 0 0) 0x7f4678142520 con 0x7f46781430c0 2024-03-25T18:23:29.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.019+0000 7f46767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:29.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.019+0000 7f46767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:29.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.019+0000 7f46767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:29.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.019+0000 7f46767fc640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:23:29.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.019+0000 7f46767fc640 1 -- 172.21.15.122:0/2307620657 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46780b28a0 con 0x7f46781430c0 2024-03-25T18:23:29.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.020+0000 7f46767fc640 1 -- 172.21.15.122:0/2307620657 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3921618893 0 0) 0x7f46780b2a30 con 0x7f46780a98f0 2024-03-25T18:23:29.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.020+0000 7f46767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:29.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.020+0000 7f46767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:29.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.020+0000 7f46767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:29.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.020+0000 7f46767fc640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:23:29.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.020+0000 7f46767fc640 1 -- 172.21.15.122:0/2307620657 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4678142520 con 0x7f46780a98f0 2024-03-25T18:23:29.023 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f46767fc640 1 -- 172.21.15.122:0/2307620657 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2856753658 0 0) 0x7f4664003520 con 0x7f467813f710 2024-03-25T18:23:29.023 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f46767fc640 1 -- 172.21.15.122:0/2307620657 >> v1:172.21.15.122:6790/0 conn(0x7f46780a98f0 legacy=0x7f46780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:29.023 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f46767fc640 1 -- 172.21.15.122:0/2307620657 >> v1:172.21.15.122:6789/0 conn(0x7f46781430c0 legacy=0x7f467814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:29.023 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f46767fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:29.023 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f46767fc640 1 monclient: found mon.noname-c 2024-03-25T18:23:29.023 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f46767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:29.023 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f46767fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:23:29.023 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f46767fc640 1 -- 172.21.15.122:0/2307620657 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46780b44c0 con 0x7f467813f710 2024-03-25T18:23:29.023 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f46767fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:29.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f46767fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:29.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f46767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:29.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f46767fc640 1 -- 172.21.15.122:0/2307620657 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4660001690 con 0x7f467813f710 2024-03-25T18:23:29.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f467d9fb640 5 monclient: authenticate success, global_id 4537 2024-03-25T18:23:29.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f467d9fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:29.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f46767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:29.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f46767fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:29.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f46767fc640 10 monclient: dump: 2024-03-25T18:23:29.024 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:29.024 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:29.024 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:29.024 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:29.024 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:29.024 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:29.024 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:29.024 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:29.025 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:29.025 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:29.025 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.021+0000 7f467d9fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:29.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.034+0000 7f46767fc640 1 -- 172.21.15.122:0/2307620657 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46600017f0 con 0x7f467813f710 2024-03-25T18:23:29.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.034+0000 7f46767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:29.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.034+0000 7f46767fc640 1 -- 172.21.15.122:0/2307620657 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4660001c00 con 0x7f467813f710 2024-03-25T18:23:29.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.034+0000 7f46767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:29.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.034+0000 7f46767fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:29.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.034+0000 7f46767fc640 10 monclient: dump: 2024-03-25T18:23:29.035 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:29.035 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:29.035 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:29.035 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:29.036 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:29.036 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:29.036 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:29.036 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:29.036 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:29.036 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:29.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.034+0000 7f467d9fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:29.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.034+0000 7f467d9fb640 4 set_mon_vals no callback set 2024-03-25T18:23:29.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.034+0000 7f467d9fb640 10 monclient: shutdown 2024-03-25T18:23:29.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.034+0000 7f467d9fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:29.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.034+0000 7f467d9fb640 1 -- 172.21.15.122:0/2307620657 >> v1:172.21.15.204:6789/0 conn(0x7f467813f710 legacy=0x7f46781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:29.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.035+0000 7f467d9fb640 1 -- 172.21.15.122:0/2307620657 shutdown_connections 2024-03-25T18:23:29.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.035+0000 7f467d9fb640 1 -- 172.21.15.122:0/2307620657 >> 172.21.15.122:0/2307620657 conn(0x7f46780b7ea0 msgr2=0x7f46780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:29.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.035+0000 7f467d9fb640 1 -- 172.21.15.122:0/2307620657 shutdown_connections 2024-03-25T18:23:29.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.035+0000 7f467d9fb640 1 -- 172.21.15.122:0/2307620657 wait complete. 2024-03-25T18:23:29.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.035+0000 7f467d9fb640 10 monclient: build_initial_monmap 2024-03-25T18:23:29.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.035+0000 7f467d9fb640 10 monclient: monmap: 2024-03-25T18:23:29.038 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:23:29.038 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:29.039 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:23:29.039 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:23:29.039 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:29.039 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:23:29.039 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:23:29.039 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:23:29.039 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:23:29.039 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:29.039 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.036+0000 7f467d9fb640 1 librados: starting msgr at 2024-03-25T18:23:29.039 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.036+0000 7f467d9fb640 1 librados: starting objecter 2024-03-25T18:23:29.039 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.036+0000 7f467d9fb640 1 Processor -- start 2024-03-25T18:23:29.039 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.036+0000 7f467d9fb640 1 -- start start 2024-03-25T18:23:29.039 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.036+0000 7f467d9fb640 1 librados: setting wanted keys 2024-03-25T18:23:29.039 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.036+0000 7f467d9fb640 1 librados: calling monclient init 2024-03-25T18:23:29.039 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.036+0000 7f467d9fb640 10 monclient: init 2024-03-25T18:23:29.040 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.037+0000 7f467d9fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:29.040 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.037+0000 7f467d9fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:29.040 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.037+0000 7f467d9fb640 10 monclient(hunting): picked mon.noname-c con 0x7f46780a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:23:29.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.037+0000 7f467d9fb640 10 monclient(hunting): picked mon.noname-a con 0x7f467813f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:23:29.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.037+0000 7f467d9fb640 10 monclient(hunting): picked mon.noname-b con 0x7f46781430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:23:29.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.037+0000 7f467d9fb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46781d77a0 con 0x7f467813f710 2024-03-25T18:23:29.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.037+0000 7f467d9fb640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4678138e40 con 0x7f46781430c0 2024-03-25T18:23:29.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.037+0000 7f467d9fb640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46780aba30 con 0x7f46780a98f0 2024-03-25T18:23:29.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.037+0000 7f467d9fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:29.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.037+0000 7f467d9fb640 10 monclient(hunting): authenticate will time out at 7732.303743s 2024-03-25T18:23:29.045 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.044+0000 7f4676ffd640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f46780a98f0 0x7f4678138730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:43016/0 (socket says 172.21.15.122:43016) 2024-03-25T18:23:29.045 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.044+0000 7f4676ffd640 1 -- 172.21.15.122:0/3752001844 learned_addr learned my addr 172.21.15.122:0/3752001844 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:23:29.046 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.045+0000 7f4674ff9640 10 client.?.objecter ms_handle_connect 0x7f46780a98f0 2024-03-25T18:23:29.046 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.045+0000 7f4674ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:29.049 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.048+0000 7f4674ff9640 10 client.?.objecter ms_handle_connect 0x7f467813f710 2024-03-25T18:23:29.049 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.048+0000 7f4674ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:29.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.051+0000 7f4674ff9640 10 client.?.objecter ms_handle_connect 0x7f46781430c0 2024-03-25T18:23:29.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.051+0000 7f4674ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:29.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.056+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727124058 0 0) 0x7f46780aba30 con 0x7f46780a98f0 2024-03-25T18:23:29.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.056+0000 7f4674ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:29.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.056+0000 7f4674ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:29.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.056+0000 7f4674ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:29.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.056+0000 7f4674ff9640 10 monclient(hunting): my global_id is 4511 2024-03-25T18:23:29.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.056+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4658004eb0 con 0x7f46780a98f0 2024-03-25T18:23:29.059 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.057+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418285160 0 0) 0x7f46781d77a0 con 0x7f467813f710 2024-03-25T18:23:29.059 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.057+0000 7f4674ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:29.059 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.057+0000 7f4674ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:29.059 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.057+0000 7f4674ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:29.059 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.057+0000 7f4674ff9640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:23:29.059 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.057+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46780aba30 con 0x7f467813f710 2024-03-25T18:23:29.066 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.064+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 964392819 0 0) 0x7f46780aba30 con 0x7f467813f710 2024-03-25T18:23:29.066 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.064+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46781d77a0 con 0x7f467813f710 2024-03-25T18:23:29.066 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.064+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4660001b00 con 0x7f467813f710 2024-03-25T18:23:29.066 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.064+0000 7f4674ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:29.066 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.064+0000 7f4674ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:29.066 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.064+0000 7f4674ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:29.066 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:29.067 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:29.067 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:29.067 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:29.067 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:29.067 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:29.067 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:29.067 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:29.067 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:29.067 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:29.067 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.065+0000 7f4674ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:29.071 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.070+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797865770 0 0) 0x7f4678138e40 con 0x7f46781430c0 2024-03-25T18:23:29.071 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.070+0000 7f4674ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:29.072 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.070+0000 7f4674ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:29.072 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.070+0000 7f4674ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:29.072 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.070+0000 7f4674ff9640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:23:29.072 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.070+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46780aba30 con 0x7f46781430c0 2024-03-25T18:23:29.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.080+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1661873022 0 0) 0x7f46780aba30 con 0x7f46781430c0 2024-03-25T18:23:29.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.080+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4678138e40 con 0x7f46781430c0 2024-03-25T18:23:29.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.080+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4670001bd0 con 0x7f46781430c0 2024-03-25T18:23:29.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.080+0000 7f4674ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:29.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.080+0000 7f4674ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:29.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.080+0000 7f4674ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:29.082 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:29.082 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:29.082 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:29.082 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:29.082 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:29.082 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:29.083 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:29.083 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:29.083 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:29.083 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:29.083 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.081+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1964086268 0 0) 0x7f4658004eb0 con 0x7f46780a98f0 2024-03-25T18:23:29.083 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.081+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46780aba30 con 0x7f46780a98f0 2024-03-25T18:23:29.083 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.081+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4668001a10 con 0x7f46780a98f0 2024-03-25T18:23:29.083 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.081+0000 7f4674ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:29.083 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.081+0000 7f4674ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:29.083 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.081+0000 7f4674ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:29.083 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:29.083 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:29.084 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:29.084 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:29.084 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:29.084 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:29.084 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:29.084 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:29.084 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:29.084 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:29.088 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.086+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1172898714 0 0) 0x7f46781d77a0 con 0x7f467813f710 2024-03-25T18:23:29.088 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.086+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 >> v1:172.21.15.122:6790/0 conn(0x7f46780a98f0 legacy=0x7f4678138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:29.088 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.086+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 >> v1:172.21.15.204:6789/0 conn(0x7f46781430c0 legacy=0x7f4678134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:29.088 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.086+0000 7f4674ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:23:29.088 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.086+0000 7f4674ff9640 1 monclient: found mon.a 2024-03-25T18:23:29.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.086+0000 7f4674ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:29.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.086+0000 7f4674ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:23:29.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.086+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46781d8340 con 0x7f467813f710 2024-03-25T18:23:29.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.086+0000 7f4674ff9640 10 monclient: _finish_auth 0 2024-03-25T18:23:29.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.086+0000 7f4674ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:23:29.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.086+0000 7f4674ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:29.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.086+0000 7f467d9fb640 5 monclient: authenticate success, global_id 4548 2024-03-25T18:23:29.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.086+0000 7f467d9fb640 10 monclient: _renew_subs 2024-03-25T18:23:29.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.086+0000 7f467d9fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:23:29.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.086+0000 7f467d9fb640 1 -- 172.21.15.122:0/3752001844 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4678134820 con 0x7f467813f710 2024-03-25T18:23:29.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.087+0000 7f467d9fb640 10 client.4548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:29.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.087+0000 7f467d9fb640 10 monclient: _renew_subs 2024-03-25T18:23:29.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.087+0000 7f467d9fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:23:29.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.087+0000 7f467d9fb640 1 -- 172.21.15.122:0/3752001844 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46781380c0 con 0x7f467813f710 2024-03-25T18:23:29.090 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.087+0000 7f467d9fb640 1 librados: init done 2024-03-25T18:23:29.090 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.088+0000 7f467d9fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:29.090 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.088+0000 7f467d9fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:29.090 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.088+0000 7f467d9fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:23:29.090 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.088+0000 7f467d9fb640 1 -- 172.21.15.122:0/3752001844 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46780627b0 con 0x7f467813f710 2024-03-25T18:23:29.388 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.387+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4660004a90 con 0x7f467813f710 2024-03-25T18:23:29.388 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.387+0000 7f4674ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:29.388 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.387+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4660004da0 con 0x7f467813f710 2024-03-25T18:23:29.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.387+0000 7f4674ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:29.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.387+0000 7f4674ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:29.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.387+0000 7f4674ff9640 10 monclient: dump: 2024-03-25T18:23:29.389 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:29.389 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:29.389 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:29.389 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:29.389 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:29.389 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:29.389 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:29.389 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:29.389 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:29.389 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:29.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.387+0000 7f4675ffb640 4 set_mon_vals no callback set 2024-03-25T18:23:29.390 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.388+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f466001ad70 con 0x7f467813f710 2024-03-25T18:23:29.390 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.389+0000 7f4676ffd640 10 monclient: get_auth_request con 0x7f465806e110 auth_method 0 2024-03-25T18:23:29.390 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.389+0000 7f4674ff9640 10 client.4548.objecter ms_handle_connect 0x7f465806e110 2024-03-25T18:23:29.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.393+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f46600547d0 con 0x7f467813f710 2024-03-25T18:23:29.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.393+0000 7f4674ff9640 10 client.4548.objecter ms_dispatch 0x7f4678000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:29.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.393+0000 7f4674ff9640 3 client.4548.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:29.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.393+0000 7f4674ff9640 3 client.4548.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:29.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.393+0000 7f4674ff9640 20 client.4548.objecter dump_active .. 0 homeless 2024-03-25T18:23:29.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.393+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4660050110 con 0x7f467813f710 2024-03-25T18:23:29.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.393+0000 7f4674ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:29.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.393+0000 7f4674ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:29.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.639+0000 7f467d9fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:29.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.639+0000 7f467d9fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:29.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.639+0000 7f467d9fb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:23:29.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.639+0000 7f467d9fb640 1 -- 172.21.15.122:0/3752001844 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f467814e880 con 0x7f467813f710 2024-03-25T18:23:29.668 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.667+0000 7f4674ff9640 1 -- 172.21.15.122:0/3752001844 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f4660001210 con 0x7f467813f710 2024-03-25T18:23:29.668 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.667+0000 7f4674ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:29.668 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.667+0000 7f4674ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:29.668 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:23:29.669 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:29.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.669+0000 7f467d9fb640 10 librados: watch_flush enter 2024-03-25T18:23:29.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.669+0000 7f467d9fb640 10 librados: watch_flush exit 2024-03-25T18:23:29.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.669+0000 7f467d9fb640 20 client.4548.objecter shutdown clearing up homeless session... 2024-03-25T18:23:29.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.669+0000 7f467d9fb640 10 client.4548.objecter successfully canceled tick 2024-03-25T18:23:29.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.669+0000 7f467d9fb640 1 -- 172.21.15.122:0/3752001844 >> v1:172.21.15.204:6800/676228005 conn(0x7f465806e110 legacy=0x7f46580705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:29.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.669+0000 7f467d9fb640 10 monclient: shutdown 2024-03-25T18:23:29.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.669+0000 7f467d9fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:29.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.669+0000 7f467d9fb640 1 -- 172.21.15.122:0/3752001844 >> v1:172.21.15.122:6789/0 conn(0x7f467813f710 legacy=0x7f46781d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:29.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.669+0000 7f467d9fb640 1 -- 172.21.15.122:0/3752001844 shutdown_connections 2024-03-25T18:23:29.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.669+0000 7f467d9fb640 1 -- 172.21.15.122:0/3752001844 >> 172.21.15.122:0/3752001844 conn(0x7f46780b7ea0 msgr2=0x7f4678074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:29.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.669+0000 7f467d9fb640 1 -- 172.21.15.122:0/3752001844 shutdown_connections 2024-03-25T18:23:29.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.669+0000 7f467d9fb640 1 -- 172.21.15.122:0/3752001844 wait complete. 2024-03-25T18:23:29.672 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:29.669+0000 7f467d9fb640 1 librados: shutdown 2024-03-25T18:23:29.682 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:36.489 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:36.603 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.602+0000 7f6005972640 10 monclient: get_monmap_and_config 2024-03-25T18:23:36.604 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.602+0000 7f6005972640 10 monclient: build_initial_monmap 2024-03-25T18:23:36.604 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.602+0000 7f6005972640 10 monclient: monmap: 2024-03-25T18:23:36.604 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:23:36.604 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:36.604 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:23:36.603819+0000 2024-03-25T18:23:36.604 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:23:36.603819+0000 2024-03-25T18:23:36.604 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:36.604 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:36.604 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:23:36.604 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:23:36.604 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:23:36.604 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:36.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.603+0000 7f6005972640 1 Processor -- start 2024-03-25T18:23:36.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.603+0000 7f6005972640 1 -- start start 2024-03-25T18:23:36.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.603+0000 7f6005972640 10 monclient: init 2024-03-25T18:23:36.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.604+0000 7f6005972640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:36.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.604+0000 7f6005972640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:36.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.604+0000 7f6005972640 10 monclient(hunting): picked mon.noname-c con 0x7f60000a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:23:36.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.604+0000 7f6005972640 10 monclient(hunting): picked mon.noname-a con 0x7f600013f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:23:36.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.604+0000 7f6005972640 10 monclient(hunting): picked mon.noname-b con 0x7f60001430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:23:36.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.604+0000 7f6005972640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6000142520 con 0x7f600013f710 2024-03-25T18:23:36.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.604+0000 7f6005972640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60000b28a0 con 0x7f60000a98f0 2024-03-25T18:23:36.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.604+0000 7f6005972640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60000b2a30 con 0x7f60001430c0 2024-03-25T18:23:36.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.604+0000 7f6005972640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:36.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.604+0000 7f6005972640 10 monclient(hunting): authenticate will time out at 7739.870361s 2024-03-25T18:23:36.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.612+0000 7f5fff7fe640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f60001430c0 0x7f600014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:49674/0 (socket says 172.21.15.122:49674) 2024-03-25T18:23:36.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.612+0000 7f5fff7fe640 1 -- 172.21.15.122:0/477792484 learned_addr learned my addr 172.21.15.122:0/477792484 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:23:36.625 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.623+0000 7f5ffe7fc640 1 -- 172.21.15.122:0/477792484 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1751557872 0 0) 0x7f60000b2a30 con 0x7f60001430c0 2024-03-25T18:23:36.625 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.623+0000 7f5ffe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:36.625 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.623+0000 7f5ffe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:36.625 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.623+0000 7f5ffe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:36.625 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.623+0000 7f5ffe7fc640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:23:36.625 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.623+0000 7f5ffe7fc640 1 -- 172.21.15.122:0/477792484 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fec003520 con 0x7f60001430c0 2024-03-25T18:23:36.634 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.633+0000 7f5ffe7fc640 1 -- 172.21.15.122:0/477792484 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601105851 0 0) 0x7f60000b28a0 con 0x7f60000a98f0 2024-03-25T18:23:36.634 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.633+0000 7f5ffe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:36.634 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.633+0000 7f5ffe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:36.634 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.633+0000 7f5ffe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:36.634 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.633+0000 7f5ffe7fc640 10 monclient(hunting): my global_id is 4543 2024-03-25T18:23:36.634 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.633+0000 7f5ffe7fc640 1 -- 172.21.15.122:0/477792484 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60000b2a30 con 0x7f60000a98f0 2024-03-25T18:23:36.639 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.637+0000 7f5ffe7fc640 1 -- 172.21.15.122:0/477792484 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1823809322 0 0) 0x7f5fec003520 con 0x7f60001430c0 2024-03-25T18:23:36.639 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.637+0000 7f5ffe7fc640 1 -- 172.21.15.122:0/477792484 >> v1:172.21.15.204:6789/0 conn(0x7f60000a98f0 legacy=0x7f60000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:36.639 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.637+0000 7f5ffe7fc640 1 -- 172.21.15.122:0/477792484 >> v1:172.21.15.122:6789/0 conn(0x7f600013f710 legacy=0x7f60001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:36.639 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.637+0000 7f5ffe7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:36.639 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.637+0000 7f5ffe7fc640 1 monclient: found mon.noname-b 2024-03-25T18:23:36.639 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.637+0000 7f5ffe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:36.639 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.637+0000 7f5ffe7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:23:36.639 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.637+0000 7f5ffe7fc640 1 -- 172.21.15.122:0/477792484 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60000b44c0 con 0x7f60001430c0 2024-03-25T18:23:36.639 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.637+0000 7f5ffe7fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:36.639 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.637+0000 7f5ffe7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:36.639 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.637+0000 7f5ffe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:36.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.637+0000 7f5ffe7fc640 1 -- 172.21.15.122:0/477792484 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5ff8001a10 con 0x7f60001430c0 2024-03-25T18:23:36.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.637+0000 7f5ffe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:36.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.637+0000 7f5ffe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:36.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.637+0000 7f5ffe7fc640 10 monclient: dump: 2024-03-25T18:23:36.640 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:36.640 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:36.640 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:36.640 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:36.640 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:36.640 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:36.640 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:36.640 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:36.640 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:36.640 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:36.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.637+0000 7f6005972640 5 monclient: authenticate success, global_id 4514 2024-03-25T18:23:36.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.637+0000 7f6005972640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:36.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.650+0000 7f5ffe7fc640 1 -- 172.21.15.122:0/477792484 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ff8001b70 con 0x7f60001430c0 2024-03-25T18:23:36.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.650+0000 7f5ffe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:36.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.650+0000 7f5ffe7fc640 1 -- 172.21.15.122:0/477792484 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5ff8001f80 con 0x7f60001430c0 2024-03-25T18:23:36.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.650+0000 7f5ffe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:36.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.650+0000 7f5ffe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:36.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.650+0000 7f5ffe7fc640 10 monclient: dump: 2024-03-25T18:23:36.651 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:36.652 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:36.652 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:36.652 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:36.652 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:36.652 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:36.652 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:36.652 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:36.652 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:36.652 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:36.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.650+0000 7f6005972640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:36.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.650+0000 7f6005972640 4 set_mon_vals no callback set 2024-03-25T18:23:36.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.651+0000 7f6005972640 10 monclient: shutdown 2024-03-25T18:23:36.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.651+0000 7f6005972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:36.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.651+0000 7f6005972640 1 -- 172.21.15.122:0/477792484 >> v1:172.21.15.122:6790/0 conn(0x7f60001430c0 legacy=0x7f600014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:36.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.651+0000 7f6005972640 1 -- 172.21.15.122:0/477792484 shutdown_connections 2024-03-25T18:23:36.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.651+0000 7f6005972640 1 -- 172.21.15.122:0/477792484 >> 172.21.15.122:0/477792484 conn(0x7f60000b7ea0 msgr2=0x7f60000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:36.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.651+0000 7f6005972640 1 -- 172.21.15.122:0/477792484 shutdown_connections 2024-03-25T18:23:36.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.651+0000 7f6005972640 1 -- 172.21.15.122:0/477792484 wait complete. 2024-03-25T18:23:36.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.652+0000 7f6005972640 10 monclient: build_initial_monmap 2024-03-25T18:23:36.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.652+0000 7f6005972640 10 monclient: monmap: 2024-03-25T18:23:36.654 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:23:36.654 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:36.654 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:23:36.655 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:23:36.655 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:36.655 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:23:36.655 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:23:36.655 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:23:36.655 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:23:36.655 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:36.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.652+0000 7f6005972640 1 librados: starting msgr at 2024-03-25T18:23:36.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.652+0000 7f6005972640 1 librados: starting objecter 2024-03-25T18:23:36.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.652+0000 7f6005972640 1 Processor -- start 2024-03-25T18:23:36.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.652+0000 7f6005972640 1 -- start start 2024-03-25T18:23:36.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.652+0000 7f6005972640 1 librados: setting wanted keys 2024-03-25T18:23:36.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.652+0000 7f6005972640 1 librados: calling monclient init 2024-03-25T18:23:36.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.652+0000 7f6005972640 10 monclient: init 2024-03-25T18:23:36.656 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.653+0000 7f6005972640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:36.656 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.653+0000 7f6005972640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:36.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.653+0000 7f6005972640 10 monclient(hunting): picked mon.noname-b con 0x7f60000a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:23:36.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.653+0000 7f6005972640 10 monclient(hunting): picked mon.noname-a con 0x7f600013f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:23:36.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.653+0000 7f6005972640 10 monclient(hunting): picked mon.noname-c con 0x7f60001430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:23:36.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.653+0000 7f6005972640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6000137ab0 con 0x7f600013f710 2024-03-25T18:23:36.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.653+0000 7f6005972640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6000138bd0 con 0x7f60000a98f0 2024-03-25T18:23:36.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.653+0000 7f6005972640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6000138e40 con 0x7f60001430c0 2024-03-25T18:23:36.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.653+0000 7f6005972640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:36.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.653+0000 7f6005972640 10 monclient(hunting): authenticate will time out at 7739.919646s 2024-03-25T18:23:36.660 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.659+0000 7f5ffeffd640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f60000a98f0 0x7f6000137f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:43430/0 (socket says 172.21.15.122:43430) 2024-03-25T18:23:36.660 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.659+0000 7f5ffeffd640 1 -- 172.21.15.122:0/4278466187 learned_addr learned my addr 172.21.15.122:0/4278466187 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:23:36.662 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.661+0000 7f5ffcff9640 10 client.?.objecter ms_handle_connect 0x7f60000a98f0 2024-03-25T18:23:36.662 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.661+0000 7f5ffcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:36.667 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.665+0000 7f5ffcff9640 10 client.?.objecter ms_handle_connect 0x7f60001430c0 2024-03-25T18:23:36.667 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.665+0000 7f5ffcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:36.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.672+0000 7f5ffcff9640 10 client.?.objecter ms_handle_connect 0x7f600013f710 2024-03-25T18:23:36.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.672+0000 7f5ffcff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:36.675 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.674+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177663284 0 0) 0x7f6000138e40 con 0x7f60001430c0 2024-03-25T18:23:36.675 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.674+0000 7f5ffcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:36.675 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.674+0000 7f5ffcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:36.675 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.674+0000 7f5ffcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:36.675 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.674+0000 7f5ffcff9640 10 monclient(hunting): my global_id is 4517 2024-03-25T18:23:36.675 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.674+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe0004eb0 con 0x7f60001430c0 2024-03-25T18:23:36.677 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.676+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066059221 0 0) 0x7f6000138bd0 con 0x7f60000a98f0 2024-03-25T18:23:36.678 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.676+0000 7f5ffcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:36.678 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.676+0000 7f5ffcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:36.678 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.676+0000 7f5ffcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:36.678 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.676+0000 7f5ffcff9640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:23:36.678 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.676+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6000138e40 con 0x7f60000a98f0 2024-03-25T18:23:36.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.689+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1865365752 0 0) 0x7f6000138e40 con 0x7f60000a98f0 2024-03-25T18:23:36.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.689+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6000138bd0 con 0x7f60000a98f0 2024-03-25T18:23:36.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.689+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5ff00019f0 con 0x7f60000a98f0 2024-03-25T18:23:36.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.690+0000 7f5ffcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:36.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.690+0000 7f5ffcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:36.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.690+0000 7f5ffcff9640 10 monclient(hunting): dump: 2024-03-25T18:23:36.691 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:36.691 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:36.691 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:36.692 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:36.692 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:36.692 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:36.692 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:36.692 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:36.692 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:36.692 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:36.692 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.690+0000 7f5ffcff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:36.692 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.690+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1094010471 0 0) 0x7f6000137ab0 con 0x7f600013f710 2024-03-25T18:23:36.693 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.690+0000 7f5ffcff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:36.693 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.690+0000 7f5ffcff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:36.693 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.690+0000 7f5ffcff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:36.693 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.690+0000 7f5ffcff9640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:23:36.693 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.690+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6000138e40 con 0x7f600013f710 2024-03-25T18:23:36.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.696+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 611537173 0 0) 0x7f5fe0004eb0 con 0x7f60001430c0 2024-03-25T18:23:36.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.696+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6000137ab0 con 0x7f60001430c0 2024-03-25T18:23:36.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.696+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5ff8001ec0 con 0x7f60001430c0 2024-03-25T18:23:36.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.696+0000 7f5ffcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:36.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.696+0000 7f5ffcff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:36.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.696+0000 7f5ffcff9640 10 monclient(hunting): dump: 2024-03-25T18:23:36.697 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:36.697 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:36.697 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:36.698 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:36.698 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:36.698 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:36.698 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:36.698 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:36.698 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:36.698 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:36.700 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.698+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3120487809 0 0) 0x7f6000138bd0 con 0x7f60000a98f0 2024-03-25T18:23:36.700 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.698+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 >> v1:172.21.15.122:6790/0 conn(0x7f60001430c0 legacy=0x7f60001e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:36.700 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.698+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 >> v1:172.21.15.122:6789/0 conn(0x7f600013f710 legacy=0x7f60001df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:36.700 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.698+0000 7f5ffcff9640 10 monclient: _finish_hunting 0 2024-03-25T18:23:36.700 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.698+0000 7f5ffcff9640 1 monclient: found mon.b 2024-03-25T18:23:36.700 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.698+0000 7f5ffcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:36.700 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.698+0000 7f5ffcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:36.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.698+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6000139120 con 0x7f60000a98f0 2024-03-25T18:23:36.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.698+0000 7f5ffcff9640 10 monclient: _finish_auth 0 2024-03-25T18:23:36.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.698+0000 7f5ffcff9640 10 monclient: _check_auth_tickets 2024-03-25T18:23:36.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.698+0000 7f5ffcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:36.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.698+0000 7f6005972640 5 monclient: authenticate success, global_id 4546 2024-03-25T18:23:36.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.698+0000 7f6005972640 10 monclient: _renew_subs 2024-03-25T18:23:36.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.698+0000 7f6005972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:36.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.699+0000 7f6005972640 1 -- 172.21.15.122:0/4278466187 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6000139400 con 0x7f60000a98f0 2024-03-25T18:23:36.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.699+0000 7f6005972640 10 client.4546.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:36.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.699+0000 7f6005972640 10 monclient: _renew_subs 2024-03-25T18:23:36.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.699+0000 7f6005972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:36.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.699+0000 7f6005972640 1 -- 172.21.15.122:0/4278466187 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60000b7880 con 0x7f60000a98f0 2024-03-25T18:23:36.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.699+0000 7f6005972640 1 librados: init done 2024-03-25T18:23:36.702 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.700+0000 7f6005972640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:36.702 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.700+0000 7f6005972640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:36.702 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.700+0000 7f6005972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:36.702 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.700+0000 7f6005972640 1 -- 172.21.15.122:0/4278466187 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6000137810 con 0x7f60000a98f0 2024-03-25T18:23:36.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.970+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ff00035f0 con 0x7f60000a98f0 2024-03-25T18:23:36.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.970+0000 7f5ffcff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:36.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.970+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5ff0003980 con 0x7f60000a98f0 2024-03-25T18:23:36.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.970+0000 7f5ffcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:36.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.970+0000 7f5ffcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:36.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.970+0000 7f5ffcff9640 10 monclient: dump: 2024-03-25T18:23:36.972 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:36.972 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:36.972 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:36.973 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:36.973 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:36.973 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:36.973 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:36.973 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:36.973 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:36.973 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:36.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.970+0000 7f5ffdffb640 4 set_mon_vals no callback set 2024-03-25T18:23:36.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.971+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f5ff0003ba0 con 0x7f60000a98f0 2024-03-25T18:23:36.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.972+0000 7f5ff7fff640 10 monclient: get_auth_request con 0x7f5fe006dfb0 auth_method 0 2024-03-25T18:23:36.974 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.972+0000 7f5ffcff9640 10 client.4546.objecter ms_handle_connect 0x7f5fe006dfb0 2024-03-25T18:23:36.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.973+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f5ff0054790 con 0x7f60000a98f0 2024-03-25T18:23:36.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.973+0000 7f5ffcff9640 10 client.4546.objecter ms_dispatch 0x7f6000000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:36.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.973+0000 7f5ffcff9640 3 client.4546.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:36.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.973+0000 7f5ffcff9640 3 client.4546.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:36.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.973+0000 7f5ffcff9640 20 client.4546.objecter dump_active .. 0 homeless 2024-03-25T18:23:36.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.979+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ff00502a0 con 0x7f60000a98f0 2024-03-25T18:23:36.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.979+0000 7f5ffcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:36.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:36.979+0000 7f5ffcff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:37.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.206+0000 7f6005972640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:37.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.206+0000 7f6005972640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:37.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.206+0000 7f6005972640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:37.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.206+0000 7f6005972640 1 -- 172.21.15.122:0/4278466187 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60000627b0 con 0x7f60000a98f0 2024-03-25T18:23:37.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.329+0000 7f5ffcff9640 1 -- 172.21.15.122:0/4278466187 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f5ff00521d0 con 0x7f60000a98f0 2024-03-25T18:23:37.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.329+0000 7f5ffcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:37.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.329+0000 7f5ffcff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:37.331 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:23:37.331 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:37.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.332+0000 7f6005972640 10 librados: watch_flush enter 2024-03-25T18:23:37.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.332+0000 7f6005972640 10 librados: watch_flush exit 2024-03-25T18:23:37.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.332+0000 7f6005972640 20 client.4546.objecter shutdown clearing up homeless session... 2024-03-25T18:23:37.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.332+0000 7f6005972640 10 client.4546.objecter successfully canceled tick 2024-03-25T18:23:37.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.332+0000 7f6005972640 1 -- 172.21.15.122:0/4278466187 >> v1:172.21.15.204:6800/676228005 conn(0x7f5fe006dfb0 legacy=0x7f5fe0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:37.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.332+0000 7f6005972640 10 monclient: shutdown 2024-03-25T18:23:37.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.332+0000 7f6005972640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:37.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.332+0000 7f6005972640 1 -- 172.21.15.122:0/4278466187 >> v1:172.21.15.204:6789/0 conn(0x7f60000a98f0 legacy=0x7f6000137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:37.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.332+0000 7f6005972640 1 -- 172.21.15.122:0/4278466187 shutdown_connections 2024-03-25T18:23:37.335 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.332+0000 7f6005972640 1 -- 172.21.15.122:0/4278466187 >> 172.21.15.122:0/4278466187 conn(0x7f60000b7ea0 msgr2=0x7f6000074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:37.335 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.332+0000 7f6005972640 1 -- 172.21.15.122:0/4278466187 shutdown_connections 2024-03-25T18:23:37.335 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.332+0000 7f6005972640 1 -- 172.21.15.122:0/4278466187 wait complete. 2024-03-25T18:23:37.335 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:37.332+0000 7f6005972640 1 librados: shutdown 2024-03-25T18:23:37.345 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:44.151 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:44.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.262+0000 7ff701b3b640 10 monclient: get_monmap_and_config 2024-03-25T18:23:44.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.262+0000 7ff701b3b640 10 monclient: build_initial_monmap 2024-03-25T18:23:44.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.262+0000 7ff701b3b640 10 monclient: monmap: 2024-03-25T18:23:44.263 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:23:44.263 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:44.263 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:23:44.263501+0000 2024-03-25T18:23:44.263 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:23:44.263501+0000 2024-03-25T18:23:44.264 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:44.264 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:44.264 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:23:44.264 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:23:44.264 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:23:44.264 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:44.264 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.263+0000 7ff701b3b640 1 Processor -- start 2024-03-25T18:23:44.265 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.263+0000 7ff701b3b640 1 -- start start 2024-03-25T18:23:44.265 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.263+0000 7ff701b3b640 10 monclient: init 2024-03-25T18:23:44.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.264+0000 7ff701b3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:44.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.264+0000 7ff701b3b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:44.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.264+0000 7ff701b3b640 10 monclient(hunting): picked mon.noname-b con 0x7ff6fc149f50 addr v1:172.21.15.122:6790/0 2024-03-25T18:23:44.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.264+0000 7ff701b3b640 10 monclient(hunting): picked mon.noname-a con 0x7ff6fc06c780 addr v1:172.21.15.122:6789/0 2024-03-25T18:23:44.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.264+0000 7ff701b3b640 10 monclient(hunting): picked mon.noname-c con 0x7ff6fc0b1b10 addr v1:172.21.15.204:6789/0 2024-03-25T18:23:44.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.264+0000 7ff701b3b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6fc14a330 con 0x7ff6fc06c780 2024-03-25T18:23:44.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.264+0000 7ff701b3b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6fc070660 con 0x7ff6fc0b1b10 2024-03-25T18:23:44.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.264+0000 7ff701b3b640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6fc06d7c0 con 0x7ff6fc149f50 2024-03-25T18:23:44.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.264+0000 7ff701b3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:44.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.264+0000 7ff701b3b640 10 monclient(hunting): authenticate will time out at 7747.530827s 2024-03-25T18:23:44.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.278+0000 7ff6fbfff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7ff6fc0b1b10 0x7ff6fc152080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:58094/0 (socket says 172.21.15.122:58094) 2024-03-25T18:23:44.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.278+0000 7ff6fbfff640 1 -- 172.21.15.122:0/3470406759 learned_addr learned my addr 172.21.15.122:0/3470406759 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:23:44.295 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.294+0000 7ff6fa7fc640 1 -- 172.21.15.122:0/3470406759 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685677934 0 0) 0x7ff6fc070660 con 0x7ff6fc0b1b10 2024-03-25T18:23:44.295 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.294+0000 7ff6fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:44.295 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.294+0000 7ff6fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:44.295 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.294+0000 7ff6fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:44.295 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.294+0000 7ff6fa7fc640 10 monclient(hunting): my global_id is 4549 2024-03-25T18:23:44.296 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.294+0000 7ff6fa7fc640 1 -- 172.21.15.122:0/3470406759 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e4003520 con 0x7ff6fc0b1b10 2024-03-25T18:23:44.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.296+0000 7ff6fa7fc640 1 -- 172.21.15.122:0/3470406759 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2878390580 0 0) 0x7ff6fc14a330 con 0x7ff6fc06c780 2024-03-25T18:23:44.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.296+0000 7ff6fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:44.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.296+0000 7ff6fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:44.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.296+0000 7ff6fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:44.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.296+0000 7ff6fa7fc640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:23:44.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.296+0000 7ff6fa7fc640 1 -- 172.21.15.122:0/3470406759 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6fc070660 con 0x7ff6fc06c780 2024-03-25T18:23:44.310 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.309+0000 7ff6fa7fc640 1 -- 172.21.15.122:0/3470406759 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2126107874 0 0) 0x7ff6fc06d7c0 con 0x7ff6fc149f50 2024-03-25T18:23:44.311 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.309+0000 7ff6fa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:44.311 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.309+0000 7ff6fa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:44.311 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.309+0000 7ff6fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:44.311 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.309+0000 7ff6fa7fc640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:23:44.311 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.309+0000 7ff6fa7fc640 1 -- 172.21.15.122:0/3470406759 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6fc14a330 con 0x7ff6fc149f50 2024-03-25T18:23:44.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.320+0000 7ff6fa7fc640 1 -- 172.21.15.122:0/3470406759 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4123851383 0 0) 0x7ff6e4003520 con 0x7ff6fc0b1b10 2024-03-25T18:23:44.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.320+0000 7ff6fa7fc640 1 -- 172.21.15.122:0/3470406759 >> v1:172.21.15.122:6790/0 conn(0x7ff6fc149f50 legacy=0x7ff6fc0b0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:44.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.320+0000 7ff6fa7fc640 1 -- 172.21.15.122:0/3470406759 >> v1:172.21.15.122:6789/0 conn(0x7ff6fc06c780 legacy=0x7ff6fc0b1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:44.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.320+0000 7ff6fa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:44.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.320+0000 7ff6fa7fc640 1 monclient: found mon.noname-c 2024-03-25T18:23:44.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.320+0000 7ff6fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:44.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.320+0000 7ff6fa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:23:44.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.321+0000 7ff6fa7fc640 1 -- 172.21.15.122:0/3470406759 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6fc06d3e0 con 0x7ff6fc0b1b10 2024-03-25T18:23:44.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.321+0000 7ff6fa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:44.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.321+0000 7ff6fa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:44.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.321+0000 7ff6fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:44.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.321+0000 7ff6fa7fc640 1 -- 172.21.15.122:0/3470406759 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff6f4001a10 con 0x7ff6fc0b1b10 2024-03-25T18:23:44.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.321+0000 7ff701b3b640 5 monclient: authenticate success, global_id 4549 2024-03-25T18:23:44.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.321+0000 7ff701b3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:44.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.321+0000 7ff6fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:44.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.321+0000 7ff6fa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:44.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.321+0000 7ff6fa7fc640 10 monclient: dump: 2024-03-25T18:23:44.323 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:44.323 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:44.323 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:44.323 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:44.323 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:44.324 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:44.324 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:44.324 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:44.324 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:44.324 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:44.324 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.321+0000 7ff701b3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:44.343 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.342+0000 7ff6fa7fc640 1 -- 172.21.15.122:0/3470406759 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6f4001b70 con 0x7ff6fc0b1b10 2024-03-25T18:23:44.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.342+0000 7ff6fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:44.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.342+0000 7ff6fa7fc640 1 -- 172.21.15.122:0/3470406759 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff6f4001f80 con 0x7ff6fc0b1b10 2024-03-25T18:23:44.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.342+0000 7ff701b3b640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:44.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.342+0000 7ff701b3b640 4 set_mon_vals no callback set 2024-03-25T18:23:44.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.342+0000 7ff6fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:44.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.342+0000 7ff6fa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:44.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.342+0000 7ff6fa7fc640 10 monclient: dump: 2024-03-25T18:23:44.344 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:44.344 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:44.344 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:44.344 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:44.344 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:44.345 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:44.345 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:44.345 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:44.345 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:44.345 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:44.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.343+0000 7ff701b3b640 10 monclient: shutdown 2024-03-25T18:23:44.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.343+0000 7ff701b3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:44.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.343+0000 7ff701b3b640 1 -- 172.21.15.122:0/3470406759 >> v1:172.21.15.204:6789/0 conn(0x7ff6fc0b1b10 legacy=0x7ff6fc152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:44.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.343+0000 7ff701b3b640 1 -- 172.21.15.122:0/3470406759 shutdown_connections 2024-03-25T18:23:44.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.343+0000 7ff701b3b640 1 -- 172.21.15.122:0/3470406759 >> 172.21.15.122:0/3470406759 conn(0x7ff6fc0b5d10 msgr2=0x7ff6fc0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:44.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.343+0000 7ff701b3b640 1 -- 172.21.15.122:0/3470406759 shutdown_connections 2024-03-25T18:23:44.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.343+0000 7ff701b3b640 1 -- 172.21.15.122:0/3470406759 wait complete. 2024-03-25T18:23:44.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.344+0000 7ff701b3b640 10 monclient: build_initial_monmap 2024-03-25T18:23:44.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.344+0000 7ff701b3b640 10 monclient: monmap: 2024-03-25T18:23:44.348 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:23:44.348 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:44.348 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:23:44.348 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:23:44.348 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:44.348 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:23:44.348 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:23:44.348 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:23:44.348 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:23:44.348 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:44.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.344+0000 7ff701b3b640 1 librados: starting msgr at 2024-03-25T18:23:44.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.344+0000 7ff701b3b640 1 librados: starting objecter 2024-03-25T18:23:44.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.345+0000 7ff701b3b640 1 Processor -- start 2024-03-25T18:23:44.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.345+0000 7ff701b3b640 1 -- start start 2024-03-25T18:23:44.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.345+0000 7ff701b3b640 1 librados: setting wanted keys 2024-03-25T18:23:44.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.345+0000 7ff701b3b640 1 librados: calling monclient init 2024-03-25T18:23:44.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.345+0000 7ff701b3b640 10 monclient: init 2024-03-25T18:23:44.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.345+0000 7ff701b3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:44.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.345+0000 7ff701b3b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:44.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.346+0000 7ff701b3b640 10 monclient(hunting): picked mon.noname-c con 0x7ff6fc06c780 addr v1:172.21.15.122:6790/0 2024-03-25T18:23:44.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.346+0000 7ff701b3b640 10 monclient(hunting): picked mon.noname-a con 0x7ff6fc0b1b10 addr v1:172.21.15.122:6789/0 2024-03-25T18:23:44.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.346+0000 7ff701b3b640 10 monclient(hunting): picked mon.noname-b con 0x7ff6fc149f50 addr v1:172.21.15.204:6789/0 2024-03-25T18:23:44.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.346+0000 7ff701b3b640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6fc0adc40 con 0x7ff6fc0b1b10 2024-03-25T18:23:44.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.346+0000 7ff701b3b640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6fc142980 con 0x7ff6fc149f50 2024-03-25T18:23:44.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.346+0000 7ff701b3b640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6fc1e00d0 con 0x7ff6fc06c780 2024-03-25T18:23:44.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.346+0000 7ff701b3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:44.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.346+0000 7ff701b3b640 10 monclient(hunting): authenticate will time out at 7747.612085s 2024-03-25T18:23:44.365 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.364+0000 7ff6faffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff6fc0b1b10 0x7ff6fc0ae0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51980/0 (socket says 172.21.15.122:51980) 2024-03-25T18:23:44.365 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.364+0000 7ff6faffd640 1 -- 172.21.15.122:0/32641888 learned_addr learned my addr 172.21.15.122:0/32641888 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:23:44.367 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.366+0000 7ff6e3fff640 10 client.?.objecter ms_handle_connect 0x7ff6fc0b1b10 2024-03-25T18:23:44.367 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.366+0000 7ff6e3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:44.371 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.370+0000 7ff6e3fff640 10 client.?.objecter ms_handle_connect 0x7ff6fc06c780 2024-03-25T18:23:44.371 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.370+0000 7ff6e3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:44.375 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.374+0000 7ff6e3fff640 10 client.?.objecter ms_handle_connect 0x7ff6fc149f50 2024-03-25T18:23:44.375 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.374+0000 7ff6e3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:44.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.385+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760670371 0 0) 0x7ff6fc142980 con 0x7ff6fc149f50 2024-03-25T18:23:44.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.385+0000 7ff6e3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:44.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.385+0000 7ff6e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:44.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.385+0000 7ff6e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:44.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.385+0000 7ff6e3fff640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:23:44.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.385+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e8004eb0 con 0x7ff6fc149f50 2024-03-25T18:23:44.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.388+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785412086 0 0) 0x7ff6fc0adc40 con 0x7ff6fc0b1b10 2024-03-25T18:23:44.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.388+0000 7ff6e3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:44.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.388+0000 7ff6e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:44.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.388+0000 7ff6e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:44.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.388+0000 7ff6e3fff640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:23:44.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.388+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6fc142980 con 0x7ff6fc0b1b10 2024-03-25T18:23:44.390 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.389+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743200791 0 0) 0x7ff6fc1e00d0 con 0x7ff6fc06c780 2024-03-25T18:23:44.390 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.389+0000 7ff6e3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:23:44.391 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.389+0000 7ff6e3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:44.391 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.389+0000 7ff6e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:44.391 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.389+0000 7ff6e3fff640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:23:44.391 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.389+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6fc0adc40 con 0x7ff6fc06c780 2024-03-25T18:23:44.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.414+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4238851969 0 0) 0x7ff6fc0adc40 con 0x7ff6fc06c780 2024-03-25T18:23:44.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.414+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6fc1e00d0 con 0x7ff6fc06c780 2024-03-25T18:23:44.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.414+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff6ec001790 con 0x7ff6fc06c780 2024-03-25T18:23:44.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.414+0000 7ff6e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:44.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.414+0000 7ff6e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:44.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.414+0000 7ff6e3fff640 10 monclient(hunting): dump: 2024-03-25T18:23:44.416 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:44.416 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:44.416 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:44.416 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:44.416 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:44.416 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:44.417 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:44.417 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:44.417 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:44.417 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:44.417 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.414+0000 7ff6e3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:44.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.428+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2688498317 0 0) 0x7ff6e8004eb0 con 0x7ff6fc149f50 2024-03-25T18:23:44.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.428+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6fc0adc40 con 0x7ff6fc149f50 2024-03-25T18:23:44.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.428+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff6f4001e80 con 0x7ff6fc149f50 2024-03-25T18:23:44.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.428+0000 7ff6e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:44.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.428+0000 7ff6e3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:44.430 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.428+0000 7ff6e3fff640 10 monclient(hunting): dump: 2024-03-25T18:23:44.430 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:44.430 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:44.430 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:44.430 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:44.431 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:44.431 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:44.431 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:44.431 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:44.431 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:44.431 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:44.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.430+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 304445739 0 0) 0x7ff6fc1e00d0 con 0x7ff6fc06c780 2024-03-25T18:23:44.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.430+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 >> v1:172.21.15.204:6789/0 conn(0x7ff6fc149f50 legacy=0x7ff6fc0ae7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:44.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 >> v1:172.21.15.122:6789/0 conn(0x7ff6fc0b1b10 legacy=0x7ff6fc0ae0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:44.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff6e3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:23:44.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff6e3fff640 1 monclient: found mon.c 2024-03-25T18:23:44.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff6e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:44.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff6e3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:23:44.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6fc1e0260 con 0x7ff6fc06c780 2024-03-25T18:23:44.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff6e3fff640 10 monclient: _finish_auth 0 2024-03-25T18:23:44.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff6e3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:23:44.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff6e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:44.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff701b3b640 5 monclient: authenticate success, global_id 4523 2024-03-25T18:23:44.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff701b3b640 10 monclient: _renew_subs 2024-03-25T18:23:44.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff701b3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:23:44.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff701b3b640 1 -- 172.21.15.122:0/32641888 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6fc1e04b0 con 0x7ff6fc06c780 2024-03-25T18:23:44.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff701b3b640 10 client.4523.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:44.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff701b3b640 10 monclient: _renew_subs 2024-03-25T18:23:44.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff701b3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:23:44.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff701b3b640 1 -- 172.21.15.122:0/32641888 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6fc1e7f50 con 0x7ff6fc06c780 2024-03-25T18:23:44.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.431+0000 7ff701b3b640 1 librados: init done 2024-03-25T18:23:44.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.432+0000 7ff701b3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:44.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.432+0000 7ff701b3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:44.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.432+0000 7ff701b3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:23:44.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.432+0000 7ff701b3b640 1 -- 172.21.15.122:0/32641888 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6fc1e06a0 con 0x7ff6fc06c780 2024-03-25T18:23:44.736 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.735+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6ec001ca0 con 0x7ff6fc06c780 2024-03-25T18:23:44.736 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.735+0000 7ff6e3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:44.736 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.735+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff6ec0037b0 con 0x7ff6fc06c780 2024-03-25T18:23:44.736 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.735+0000 7ff6e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:44.736 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.735+0000 7ff6e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:44.736 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.735+0000 7ff6e3fff640 10 monclient: dump: 2024-03-25T18:23:44.736 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:44.737 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:44.737 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:44.737 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:44.737 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:44.737 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:44.737 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:44.737 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:44.737 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:44.737 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:44.737 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.735+0000 7ff6f9ffb640 4 set_mon_vals no callback set 2024-03-25T18:23:44.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.736+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7ff6ec019780 con 0x7ff6fc06c780 2024-03-25T18:23:44.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.736+0000 7ff6faffd640 10 monclient: get_auth_request con 0x7ff6e806df20 auth_method 0 2024-03-25T18:23:44.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.736+0000 7ff6e3fff640 10 client.4523.objecter ms_handle_connect 0x7ff6e806df20 2024-03-25T18:23:44.743 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.742+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7ff6ec0534e0 con 0x7ff6fc06c780 2024-03-25T18:23:44.743 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.742+0000 7ff6e3fff640 10 client.4523.objecter ms_dispatch 0x7ff6fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:44.743 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.742+0000 7ff6e3fff640 3 client.4523.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:44.743 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.742+0000 7ff6e3fff640 3 client.4523.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:44.743 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.742+0000 7ff6e3fff640 20 client.4523.objecter dump_active .. 0 homeless 2024-03-25T18:23:44.743 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.742+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6ec04ee20 con 0x7ff6fc06c780 2024-03-25T18:23:44.743 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.742+0000 7ff6e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:44.743 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.742+0000 7ff6e3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:44.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.967+0000 7ff701b3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:44.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.967+0000 7ff701b3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:44.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.967+0000 7ff701b3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:23:44.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:44.967+0000 7ff701b3b640 1 -- 172.21.15.122:0/32641888 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6fc0b15e0 con 0x7ff6fc06c780 2024-03-25T18:23:45.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:45.072+0000 7ff6e3fff640 1 -- 172.21.15.122:0/32641888 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7ff6ec050f20 con 0x7ff6fc06c780 2024-03-25T18:23:45.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:45.072+0000 7ff6e3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:45.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:45.072+0000 7ff6e3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:45.074 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:23:45.074 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:45.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:45.075+0000 7ff701b3b640 10 librados: watch_flush enter 2024-03-25T18:23:45.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:45.075+0000 7ff701b3b640 10 librados: watch_flush exit 2024-03-25T18:23:45.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:45.075+0000 7ff701b3b640 20 client.4523.objecter shutdown clearing up homeless session... 2024-03-25T18:23:45.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:45.075+0000 7ff701b3b640 10 client.4523.objecter successfully canceled tick 2024-03-25T18:23:45.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:45.075+0000 7ff701b3b640 1 -- 172.21.15.122:0/32641888 >> v1:172.21.15.204:6800/676228005 conn(0x7ff6e806df20 legacy=0x7ff6e80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:45.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:45.075+0000 7ff701b3b640 10 monclient: shutdown 2024-03-25T18:23:45.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:45.075+0000 7ff701b3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:45.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:45.075+0000 7ff701b3b640 1 -- 172.21.15.122:0/32641888 >> v1:172.21.15.122:6790/0 conn(0x7ff6fc06c780 legacy=0x7ff6fc1df9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:45.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:45.075+0000 7ff701b3b640 1 -- 172.21.15.122:0/32641888 shutdown_connections 2024-03-25T18:23:45.078 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:45.075+0000 7ff701b3b640 1 -- 172.21.15.122:0/32641888 >> 172.21.15.122:0/32641888 conn(0x7ff6fc0b5d10 msgr2=0x7ff6fc0b42d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:45.078 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:45.075+0000 7ff701b3b640 1 -- 172.21.15.122:0/32641888 shutdown_connections 2024-03-25T18:23:45.078 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:45.075+0000 7ff701b3b640 1 -- 172.21.15.122:0/32641888 wait complete. 2024-03-25T18:23:45.078 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:45.075+0000 7ff701b3b640 1 librados: shutdown 2024-03-25T18:23:45.089 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:51.896 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:51.968 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:07:45.711 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:51.983 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:07:45.726 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:51.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.996+0000 7f5a70e2e640 10 monclient: get_monmap_and_config 2024-03-25T18:23:51.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.996+0000 7f5a70e2e640 10 monclient: build_initial_monmap 2024-03-25T18:23:51.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.996+0000 7f5a70e2e640 10 monclient: monmap: 2024-03-25T18:23:51.997 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:23:51.997 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:51.997 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:23:51.997394+0000 2024-03-25T18:23:51.998 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:23:51.997394+0000 2024-03-25T18:23:51.998 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:51.998 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:51.998 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:23:51.998 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:23:51.998 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:23:51.998 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:51.999 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.996+0000 7f5a70e2e640 1 Processor -- start 2024-03-25T18:23:51.999 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.996+0000 7f5a70e2e640 1 -- start start 2024-03-25T18:23:51.999 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.996+0000 7f5a70e2e640 10 monclient: init 2024-03-25T18:23:52.000 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.997+0000 7f5a70e2e640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:52.000 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.997+0000 7f5a70e2e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:52.000 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.997+0000 7f5a70e2e640 10 monclient(hunting): picked mon.noname-c con 0x7f5a6c0b4d60 addr v1:172.21.15.204:6789/0 2024-03-25T18:23:52.000 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.997+0000 7f5a70e2e640 10 monclient(hunting): picked mon.noname-a con 0x7f5a6c0b1730 addr v1:172.21.15.122:6789/0 2024-03-25T18:23:52.000 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.997+0000 7f5a70e2e640 10 monclient(hunting): picked mon.noname-b con 0x7f5a6c14be90 addr v1:172.21.15.122:6790/0 2024-03-25T18:23:52.000 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.997+0000 7f5a70e2e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c0af940 con 0x7f5a6c0b1730 2024-03-25T18:23:52.000 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.998+0000 7f5a70e2e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c0b0a70 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.000 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.998+0000 7f5a70e2e640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c0b0d50 con 0x7f5a6c14be90 2024-03-25T18:23:52.000 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.998+0000 7f5a70e2e640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:52.000 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.998+0000 7f5a70e2e640 10 monclient(hunting): authenticate will time out at 7755.264062s 2024-03-25T18:23:52.000 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.998+0000 7f5a6a59c640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f5a6c0b4d60 0x7f5a6c13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:60328/0 (socket says 172.21.15.122:60328) 2024-03-25T18:23:52.000 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:51.998+0000 7f5a6a59c640 1 -- 172.21.15.122:0/324144791 learned_addr learned my addr 172.21.15.122:0/324144791 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:23:52.010 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:07:45.752 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:52.025 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:07:45.768 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:52.037 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:07:45.780 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:52.043 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.042+0000 7f5a6959a640 1 -- 172.21.15.122:0/324144791 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3881062840 0 0) 0x7f5a6c0af940 con 0x7f5a6c0b1730 2024-03-25T18:23:52.044 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.042+0000 7f5a6959a640 10 monclient(hunting): _negotiate 2024-03-25T18:23:52.044 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.042+0000 7f5a6959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:52.044 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.042+0000 7f5a6959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:52.044 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.042+0000 7f5a6959a640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:23:52.044 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.042+0000 7f5a6959a640 1 -- 172.21.15.122:0/324144791 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a54003520 con 0x7f5a6c0b1730 2024-03-25T18:23:52.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.047+0000 7f5a6959a640 1 -- 172.21.15.122:0/324144791 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311574691 0 0) 0x7f5a6c0b0d50 con 0x7f5a6c14be90 2024-03-25T18:23:52.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.047+0000 7f5a6959a640 10 monclient(hunting): _negotiate 2024-03-25T18:23:52.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.047+0000 7f5a6959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:52.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.047+0000 7f5a6959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:52.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.047+0000 7f5a6959a640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:23:52.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.047+0000 7f5a6959a640 1 -- 172.21.15.122:0/324144791 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a6c0af940 con 0x7f5a6c14be90 2024-03-25T18:23:52.066 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.064+0000 7f5a6959a640 1 -- 172.21.15.122:0/324144791 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3778503250 0 0) 0x7f5a6c0af940 con 0x7f5a6c14be90 2024-03-25T18:23:52.066 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.064+0000 7f5a6959a640 1 -- 172.21.15.122:0/324144791 >> v1:172.21.15.204:6789/0 conn(0x7f5a6c0b4d60 legacy=0x7f5a6c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:52.066 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.064+0000 7f5a6959a640 1 -- 172.21.15.122:0/324144791 >> v1:172.21.15.122:6789/0 conn(0x7f5a6c0b1730 legacy=0x7f5a6c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:52.066 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.064+0000 7f5a6959a640 10 monclient: _finish_hunting 0 2024-03-25T18:23:52.066 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.064+0000 7f5a6959a640 1 monclient: found mon.noname-b 2024-03-25T18:23:52.066 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.064+0000 7f5a6959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:52.066 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.064+0000 7f5a6959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:23:52.066 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.064+0000 7f5a6959a640 1 -- 172.21.15.122:0/324144791 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c0b1030 con 0x7f5a6c14be90 2024-03-25T18:23:52.066 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.064+0000 7f5a6959a640 10 monclient: _finish_auth 0 2024-03-25T18:23:52.067 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.064+0000 7f5a6959a640 10 monclient: _check_auth_tickets 2024-03-25T18:23:52.067 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.064+0000 7f5a6959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:52.067 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.064+0000 7f5a6959a640 1 -- 172.21.15.122:0/324144791 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5a640019f0 con 0x7f5a6c14be90 2024-03-25T18:23:52.067 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.064+0000 7f5a6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:52.067 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.064+0000 7f5a6959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:52.067 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.064+0000 7f5a6959a640 10 monclient: dump: 2024-03-25T18:23:52.067 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:52.067 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:52.067 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:52.067 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:52.067 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:52.067 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:52.067 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:52.068 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:52.068 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:52.068 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:52.068 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.064+0000 7f5a70e2e640 5 monclient: authenticate success, global_id 4526 2024-03-25T18:23:52.068 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.064+0000 7f5a70e2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:52.073 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:07:45.816 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:52.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.083+0000 7f5a6959a640 1 -- 172.21.15.122:0/324144791 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a64001b50 con 0x7f5a6c14be90 2024-03-25T18:23:52.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.083+0000 7f5a6959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:52.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.083+0000 7f5a6959a640 1 -- 172.21.15.122:0/324144791 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5a64001f20 con 0x7f5a6c14be90 2024-03-25T18:23:52.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.083+0000 7f5a6959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:52.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.083+0000 7f5a6959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:52.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.083+0000 7f5a6959a640 10 monclient: dump: 2024-03-25T18:23:52.085 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:52.085 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:52.085 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:52.085 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:52.085 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:52.085 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:52.085 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:52.085 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:52.085 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:52.085 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:52.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.083+0000 7f5a70e2e640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:52.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.083+0000 7f5a70e2e640 4 set_mon_vals no callback set 2024-03-25T18:23:52.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.084+0000 7f5a70e2e640 10 monclient: shutdown 2024-03-25T18:23:52.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.084+0000 7f5a70e2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:52.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.084+0000 7f5a70e2e640 1 -- 172.21.15.122:0/324144791 >> v1:172.21.15.122:6790/0 conn(0x7f5a6c14be90 legacy=0x7f5a6c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:52.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.084+0000 7f5a70e2e640 1 -- 172.21.15.122:0/324144791 shutdown_connections 2024-03-25T18:23:52.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.084+0000 7f5a70e2e640 1 -- 172.21.15.122:0/324144791 >> 172.21.15.122:0/324144791 conn(0x7f5a6c0617a0 msgr2=0x7f5a6c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:52.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.084+0000 7f5a70e2e640 1 -- 172.21.15.122:0/324144791 shutdown_connections 2024-03-25T18:23:52.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.084+0000 7f5a70e2e640 1 -- 172.21.15.122:0/324144791 wait complete. 2024-03-25T18:23:52.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.085+0000 7f5a70e2e640 10 monclient: build_initial_monmap 2024-03-25T18:23:52.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.085+0000 7f5a70e2e640 10 monclient: monmap: 2024-03-25T18:23:52.087 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:23:52.087 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:52.087 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:23:52.088 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:23:52.088 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:52.088 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:23:52.088 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:23:52.088 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:23:52.088 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:23:52.088 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:52.088 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.086+0000 7f5a70e2e640 1 librados: starting msgr at 2024-03-25T18:23:52.088 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.086+0000 7f5a70e2e640 1 librados: starting objecter 2024-03-25T18:23:52.088 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.086+0000 7f5a70e2e640 1 Processor -- start 2024-03-25T18:23:52.088 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.086+0000 7f5a70e2e640 1 -- start start 2024-03-25T18:23:52.088 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.086+0000 7f5a70e2e640 1 librados: setting wanted keys 2024-03-25T18:23:52.088 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.086+0000 7f5a70e2e640 1 librados: calling monclient init 2024-03-25T18:23:52.088 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.086+0000 7f5a70e2e640 10 monclient: init 2024-03-25T18:23:52.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.086+0000 7f5a70e2e640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:52.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.086+0000 7f5a70e2e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:52.089 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.086+0000 7f5a70e2e640 10 monclient(hunting): picked mon.noname-a con 0x7f5a6c0b1730 addr v1:172.21.15.122:6789/0 2024-03-25T18:23:52.090 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.086+0000 7f5a70e2e640 10 monclient(hunting): picked mon.noname-b con 0x7f5a6c0b4d60 addr v1:172.21.15.204:6789/0 2024-03-25T18:23:52.090 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.086+0000 7f5a70e2e640 10 monclient(hunting): picked mon.noname-c con 0x7f5a6c14be90 addr v1:172.21.15.122:6790/0 2024-03-25T18:23:52.090 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.086+0000 7f5a70e2e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c1327e0 con 0x7f5a6c0b1730 2024-03-25T18:23:52.090 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.086+0000 7f5a70e2e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c140a10 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.090 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.086+0000 7f5a70e2e640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c137eb0 con 0x7f5a6c14be90 2024-03-25T18:23:52.090 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.086+0000 7f5a70e2e640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:52.090 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.086+0000 7f5a70e2e640 10 monclient(hunting): authenticate will time out at 7755.352788s 2024-03-25T18:23:52.097 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:07:45.843 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:52.101 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.100+0000 7f5a6a59c640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5a6c0b1730 0x7f5a6c1d7830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:57628/0 (socket says 172.21.15.122:57628) 2024-03-25T18:23:52.101 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.100+0000 7f5a6a59c640 1 -- 172.21.15.122:0/3619247580 learned_addr learned my addr 172.21.15.122:0/3619247580 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:23:52.103 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:07:45.848 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:52.106 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.105+0000 7f5a5affd640 10 client.?.objecter ms_handle_connect 0x7f5a6c0b1730 2024-03-25T18:23:52.106 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.105+0000 7f5a5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:52.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.119+0000 7f5a5affd640 10 client.?.objecter ms_handle_connect 0x7f5a6c14be90 2024-03-25T18:23:52.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.119+0000 7f5a5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:52.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.120+0000 7f5a5affd640 10 client.?.objecter ms_handle_connect 0x7f5a6c0b4d60 2024-03-25T18:23:52.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.120+0000 7f5a5affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:52.129 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.128+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3916700141 0 0) 0x7f5a6c140a10 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.129 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.128+0000 7f5a5affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:52.130 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.128+0000 7f5a5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:52.130 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.128+0000 7f5a5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:52.130 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.128+0000 7f5a5affd640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:23:52.130 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.128+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a4c004eb0 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.131 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:07:45.877 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:52.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.138+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515162172 0 0) 0x7f5a6c1327e0 con 0x7f5a6c0b1730 2024-03-25T18:23:52.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.138+0000 7f5a5affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:52.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.138+0000 7f5a5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:52.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.138+0000 7f5a5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:52.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.138+0000 7f5a5affd640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:23:52.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.138+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a6c140a10 con 0x7f5a6c0b1730 2024-03-25T18:23:52.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.163+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2405232418 0 0) 0x7f5a4c004eb0 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.163+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a6c1327e0 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.163+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5a60001950 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.163+0000 7f5a5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.163+0000 7f5a5affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.163+0000 7f5a5affd640 10 monclient(hunting): dump: 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:52.165 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:52.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.163+0000 7f5a5affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:52.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.165+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955780659 0 0) 0x7f5a6c137eb0 con 0x7f5a6c14be90 2024-03-25T18:23:52.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.165+0000 7f5a5affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:52.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.165+0000 7f5a5affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:52.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.165+0000 7f5a5affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:52.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.165+0000 7f5a5affd640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:23:52.166 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.165+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a4c004eb0 con 0x7f5a6c14be90 2024-03-25T18:23:52.170 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.169+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3023075537 0 0) 0x7f5a6c140a10 con 0x7f5a6c0b1730 2024-03-25T18:23:52.170 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.169+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a6c137eb0 con 0x7f5a6c0b1730 2024-03-25T18:23:52.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.169+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5a5c0018f0 con 0x7f5a6c0b1730 2024-03-25T18:23:52.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.169+0000 7f5a5affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:52.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.169+0000 7f5a5affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:52.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.169+0000 7f5a5affd640 10 monclient(hunting): dump: 2024-03-25T18:23:52.171 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:52.171 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:52.171 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:52.171 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:52.171 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:52.171 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:52.171 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:52.171 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:52.171 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:52.171 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:52.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.181+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2559914745 0 0) 0x7f5a6c1327e0 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.181+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 >> v1:172.21.15.122:6790/0 conn(0x7f5a6c14be90 legacy=0x7f5a6c1320d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:52.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.181+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 >> v1:172.21.15.122:6789/0 conn(0x7f5a6c0b1730 legacy=0x7f5a6c1d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:52.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.181+0000 7f5a5affd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:52.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.181+0000 7f5a5affd640 1 monclient: found mon.b 2024-03-25T18:23:52.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.181+0000 7f5a5affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:52.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.182+0000 7f5a5affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:52.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.182+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c138040 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.182+0000 7f5a5affd640 10 monclient: _finish_auth 0 2024-03-25T18:23:52.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.182+0000 7f5a5affd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:52.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.182+0000 7f5a5affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:52.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.182+0000 7f5a70e2e640 5 monclient: authenticate success, global_id 4558 2024-03-25T18:23:52.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.182+0000 7f5a70e2e640 10 monclient: _renew_subs 2024-03-25T18:23:52.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.182+0000 7f5a70e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:52.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.182+0000 7f5a70e2e640 1 -- 172.21.15.122:0/3619247580 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6c140380 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.182+0000 7f5a70e2e640 10 client.4558.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:52.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.182+0000 7f5a70e2e640 10 monclient: _renew_subs 2024-03-25T18:23:52.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.182+0000 7f5a70e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:52.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.182+0000 7f5a70e2e640 1 -- 172.21.15.122:0/3619247580 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6c1452a0 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.182+0000 7f5a70e2e640 1 librados: init done 2024-03-25T18:23:52.185 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.183+0000 7f5a70e2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:52.185 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.183+0000 7f5a70e2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:52.185 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.183+0000 7f5a70e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:52.185 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.183+0000 7f5a70e2e640 1 -- 172.21.15.122:0/3619247580 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6c0b22c0 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.470+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a60003610 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.470+0000 7f5a5affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:52.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.470+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5a60003920 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.470+0000 7f5a5affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:52.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.470+0000 7f5a5affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:52.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.470+0000 7f5a5affd640 10 monclient: dump: 2024-03-25T18:23:52.471 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:52.471 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:52.472 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:52.472 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:52.472 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:52.472 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:52.472 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:52.472 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:52.472 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:52.472 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:52.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.470+0000 7f5a68d99640 4 set_mon_vals no callback set 2024-03-25T18:23:52.472 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.470+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f5a600198f0 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.471+0000 7f5a6a59c640 10 monclient: get_auth_request con 0x7f5a4c06df20 auth_method 0 2024-03-25T18:23:52.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.471+0000 7f5a5affd640 10 client.4558.objecter ms_handle_connect 0x7f5a4c06df20 2024-03-25T18:23:52.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.477+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f5a60053640 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.477+0000 7f5a5affd640 10 client.4558.objecter ms_dispatch 0x7f5a6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:52.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.477+0000 7f5a5affd640 3 client.4558.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:52.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.477+0000 7f5a5affd640 3 client.4558.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:52.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.477+0000 7f5a5affd640 20 client.4558.objecter dump_active .. 0 homeless 2024-03-25T18:23:52.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.482+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a6004ef80 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.482+0000 7f5a5affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:52.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.482+0000 7f5a5affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:52.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.737+0000 7f5a70e2e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:52.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.737+0000 7f5a70e2e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:52.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.737+0000 7f5a70e2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:52.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.737+0000 7f5a70e2e640 1 -- 172.21.15.122:0/3619247580 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a6c0627b0 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.760 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.759+0000 7f5a5affd640 1 -- 172.21.15.122:0/3619247580 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f5a60051080 con 0x7f5a6c0b4d60 2024-03-25T18:23:52.760 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.759+0000 7f5a5affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:52.760 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.759+0000 7f5a5affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:52.761 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:23:52.761 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:52.764 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.762+0000 7f5a70e2e640 10 librados: watch_flush enter 2024-03-25T18:23:52.764 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.762+0000 7f5a70e2e640 10 librados: watch_flush exit 2024-03-25T18:23:52.764 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.762+0000 7f5a70e2e640 20 client.4558.objecter shutdown clearing up homeless session... 2024-03-25T18:23:52.764 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.762+0000 7f5a70e2e640 10 client.4558.objecter successfully canceled tick 2024-03-25T18:23:52.764 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.762+0000 7f5a70e2e640 1 -- 172.21.15.122:0/3619247580 >> v1:172.21.15.204:6800/676228005 conn(0x7f5a4c06df20 legacy=0x7f5a4c0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:52.764 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.762+0000 7f5a70e2e640 10 monclient: shutdown 2024-03-25T18:23:52.764 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.762+0000 7f5a70e2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:52.764 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.762+0000 7f5a70e2e640 1 -- 172.21.15.122:0/3619247580 >> v1:172.21.15.204:6789/0 conn(0x7f5a6c0b4d60 legacy=0x7f5a6c1319c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:52.764 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.762+0000 7f5a70e2e640 1 -- 172.21.15.122:0/3619247580 shutdown_connections 2024-03-25T18:23:52.764 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.762+0000 7f5a70e2e640 1 -- 172.21.15.122:0/3619247580 >> 172.21.15.122:0/3619247580 conn(0x7f5a6c0617a0 msgr2=0x7f5a6c13c5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:52.764 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.762+0000 7f5a70e2e640 1 -- 172.21.15.122:0/3619247580 shutdown_connections 2024-03-25T18:23:52.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.762+0000 7f5a70e2e640 1 -- 172.21.15.122:0/3619247580 wait complete. 2024-03-25T18:23:52.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:52.762+0000 7f5a70e2e640 1 librados: shutdown 2024-03-25T18:23:52.775 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:59.582 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:59.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.699+0000 7f4e0a838640 10 monclient: get_monmap_and_config 2024-03-25T18:23:59.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.699+0000 7f4e0a838640 10 monclient: build_initial_monmap 2024-03-25T18:23:59.701 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.699+0000 7f4e0a838640 10 monclient: monmap: 2024-03-25T18:23:59.701 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:23:59.701 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:59.701 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:23:59.700977+0000 2024-03-25T18:23:59.701 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:23:59.700977+0000 2024-03-25T18:23:59.701 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:59.701 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:59.701 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:23:59.701 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:23:59.701 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:23:59.701 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:59.703 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.700+0000 7f4e0a838640 1 Processor -- start 2024-03-25T18:23:59.703 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.700+0000 7f4e0a838640 1 -- start start 2024-03-25T18:23:59.703 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.700+0000 7f4e0a838640 10 monclient: init 2024-03-25T18:23:59.703 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.701+0000 7f4e0a838640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:59.703 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.701+0000 7f4e0a838640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:59.703 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.701+0000 7f4e0a838640 10 monclient(hunting): picked mon.noname-b con 0x7f4e040b4d60 addr v1:172.21.15.122:6790/0 2024-03-25T18:23:59.703 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.701+0000 7f4e0a838640 10 monclient(hunting): picked mon.noname-a con 0x7f4e040b1730 addr v1:172.21.15.122:6789/0 2024-03-25T18:23:59.703 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.701+0000 7f4e0a838640 10 monclient(hunting): picked mon.noname-c con 0x7f4e0414be90 addr v1:172.21.15.204:6789/0 2024-03-25T18:23:59.703 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.701+0000 7f4e0a838640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e040af940 con 0x7f4e040b1730 2024-03-25T18:23:59.703 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.701+0000 7f4e0a838640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e040b0a70 con 0x7f4e0414be90 2024-03-25T18:23:59.703 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.701+0000 7f4e0a838640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e040b0d50 con 0x7f4e040b4d60 2024-03-25T18:23:59.703 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.701+0000 7f4e0a838640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:59.703 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.701+0000 7f4e0a838640 10 monclient(hunting): authenticate will time out at 7762.967624s 2024-03-25T18:23:59.718 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.717+0000 7f4e037fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f4e040b1730 0x7f4e040b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:54152/0 (socket says 172.21.15.122:54152) 2024-03-25T18:23:59.718 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.717+0000 7f4e037fe640 1 -- 172.21.15.122:0/1135522253 learned_addr learned my addr 172.21.15.122:0/1135522253 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:23:59.743 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.742+0000 7f4e02ffd640 1 -- 172.21.15.122:0/1135522253 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913658042 0 0) 0x7f4e040b0d50 con 0x7f4e040b4d60 2024-03-25T18:23:59.743 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.742+0000 7f4e02ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:59.744 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.742+0000 7f4e02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:59.744 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.742+0000 7f4e02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:59.744 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.742+0000 7f4e02ffd640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:23:59.744 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.742+0000 7f4e02ffd640 1 -- 172.21.15.122:0/1135522253 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4dec003520 con 0x7f4e040b4d60 2024-03-25T18:23:59.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.747+0000 7f4e02ffd640 1 -- 172.21.15.122:0/1135522253 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 149218152 0 0) 0x7f4e040af940 con 0x7f4e040b1730 2024-03-25T18:23:59.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.747+0000 7f4e02ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:59.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.747+0000 7f4e02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:59.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.747+0000 7f4e02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:59.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.747+0000 7f4e02ffd640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:23:59.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.747+0000 7f4e02ffd640 1 -- 172.21.15.122:0/1135522253 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e040b0d50 con 0x7f4e040b1730 2024-03-25T18:23:59.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.748+0000 7f4e02ffd640 1 -- 172.21.15.122:0/1135522253 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167437352 0 0) 0x7f4e040b0a70 con 0x7f4e0414be90 2024-03-25T18:23:59.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.748+0000 7f4e02ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:59.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.748+0000 7f4e02ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:59.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.748+0000 7f4e02ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:59.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.748+0000 7f4e02ffd640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:23:59.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.748+0000 7f4e02ffd640 1 -- 172.21.15.122:0/1135522253 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e040af940 con 0x7f4e0414be90 2024-03-25T18:23:59.773 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.772+0000 7f4e02ffd640 1 -- 172.21.15.122:0/1135522253 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 73542839 0 0) 0x7f4dec003520 con 0x7f4e040b4d60 2024-03-25T18:23:59.773 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.772+0000 7f4e02ffd640 1 -- 172.21.15.122:0/1135522253 >> v1:172.21.15.204:6789/0 conn(0x7f4e0414be90 legacy=0x7f4e0414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:59.773 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.772+0000 7f4e02ffd640 1 -- 172.21.15.122:0/1135522253 >> v1:172.21.15.122:6789/0 conn(0x7f4e040b1730 legacy=0x7f4e040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:59.773 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.772+0000 7f4e02ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:59.773 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.772+0000 7f4e02ffd640 1 monclient: found mon.noname-b 2024-03-25T18:23:59.773 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.772+0000 7f4e02ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:59.773 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.772+0000 7f4e02ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.772+0000 7f4e02ffd640 1 -- 172.21.15.122:0/1135522253 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e040b1030 con 0x7f4e040b4d60 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.772+0000 7f4e02ffd640 10 monclient: _finish_auth 0 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.772+0000 7f4e02ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.772+0000 7f4e02ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.772+0000 7f4e02ffd640 1 -- 172.21.15.122:0/1135522253 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4df4001690 con 0x7f4e040b4d60 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.772+0000 7f4e02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.772+0000 7f4e02ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.772+0000 7f4e02ffd640 10 monclient: dump: 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:59.774 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:59.775 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:59.775 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:59.775 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:59.775 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:59.775 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.772+0000 7f4e0a838640 5 monclient: authenticate success, global_id 4532 2024-03-25T18:23:59.775 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.772+0000 7f4e0a838640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:59.800 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.799+0000 7f4e02ffd640 1 -- 172.21.15.122:0/1135522253 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4df40017f0 con 0x7f4e040b4d60 2024-03-25T18:23:59.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.799+0000 7f4e02ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:59.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.799+0000 7f4e02ffd640 1 -- 172.21.15.122:0/1135522253 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4df4001c00 con 0x7f4e040b4d60 2024-03-25T18:23:59.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.799+0000 7f4e0a838640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:59.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.799+0000 7f4e0a838640 4 set_mon_vals no callback set 2024-03-25T18:23:59.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.799+0000 7f4e02ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:59.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.799+0000 7f4e02ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:59.801 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.799+0000 7f4e02ffd640 10 monclient: dump: 2024-03-25T18:23:59.801 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:59.801 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:59.801 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:59.801 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:59.801 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:59.801 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:59.801 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:59.801 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:59.801 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:59.802 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:59.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.800+0000 7f4e0a838640 10 monclient: shutdown 2024-03-25T18:23:59.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.800+0000 7f4e0a838640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:59.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.800+0000 7f4e0a838640 1 -- 172.21.15.122:0/1135522253 >> v1:172.21.15.122:6790/0 conn(0x7f4e040b4d60 legacy=0x7f4e0413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:59.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.800+0000 7f4e0a838640 1 -- 172.21.15.122:0/1135522253 shutdown_connections 2024-03-25T18:23:59.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.800+0000 7f4e0a838640 1 -- 172.21.15.122:0/1135522253 >> 172.21.15.122:0/1135522253 conn(0x7f4e040617a0 msgr2=0x7f4e04061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:59.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.800+0000 7f4e0a838640 1 -- 172.21.15.122:0/1135522253 shutdown_connections 2024-03-25T18:23:59.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.800+0000 7f4e0a838640 1 -- 172.21.15.122:0/1135522253 wait complete. 2024-03-25T18:23:59.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.801+0000 7f4e0a838640 10 monclient: build_initial_monmap 2024-03-25T18:23:59.803 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.801+0000 7f4e0a838640 10 monclient: monmap: 2024-03-25T18:23:59.803 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:23:59.804 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:59.804 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:23:59.804 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:23:59.804 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:59.804 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:23:59.804 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:23:59.804 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:23:59.804 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:23:59.804 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:59.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.801+0000 7f4e0a838640 1 librados: starting msgr at 2024-03-25T18:23:59.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.801+0000 7f4e0a838640 1 librados: starting objecter 2024-03-25T18:23:59.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.801+0000 7f4e0a838640 1 Processor -- start 2024-03-25T18:23:59.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.802+0000 7f4e0a838640 1 -- start start 2024-03-25T18:23:59.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.802+0000 7f4e0a838640 1 librados: setting wanted keys 2024-03-25T18:23:59.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.802+0000 7f4e0a838640 1 librados: calling monclient init 2024-03-25T18:23:59.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.802+0000 7f4e0a838640 10 monclient: init 2024-03-25T18:23:59.805 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.802+0000 7f4e0a838640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:59.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.802+0000 7f4e0a838640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:59.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.802+0000 7f4e0a838640 10 monclient(hunting): picked mon.noname-b con 0x7f4e040b1730 addr v1:172.21.15.204:6789/0 2024-03-25T18:23:59.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.802+0000 7f4e0a838640 10 monclient(hunting): picked mon.noname-a con 0x7f4e040b4d60 addr v1:172.21.15.122:6789/0 2024-03-25T18:23:59.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.802+0000 7f4e0a838640 10 monclient(hunting): picked mon.noname-c con 0x7f4e0414be90 addr v1:172.21.15.122:6790/0 2024-03-25T18:23:59.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.802+0000 7f4e0a838640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e04142d20 con 0x7f4e040b4d60 2024-03-25T18:23:59.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.802+0000 7f4e0a838640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e04134f90 con 0x7f4e040b1730 2024-03-25T18:23:59.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.802+0000 7f4e0a838640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e0413a0a0 con 0x7f4e0414be90 2024-03-25T18:23:59.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.802+0000 7f4e0a838640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:59.806 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.802+0000 7f4e0a838640 10 monclient(hunting): authenticate will time out at 7763.068920s 2024-03-25T18:23:59.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.807+0000 7f4e08dd5640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f4e0414be90 0x7f4e04134880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:58036/0 (socket says 172.21.15.122:58036) 2024-03-25T18:23:59.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.807+0000 7f4e08dd5640 1 -- 172.21.15.122:0/652970553 learned_addr learned my addr 172.21.15.122:0/652970553 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:23:59.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.808+0000 7f4e00ff9640 10 client.?.objecter ms_handle_connect 0x7f4e0414be90 2024-03-25T18:23:59.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.808+0000 7f4e00ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:59.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.812+0000 7f4e00ff9640 10 client.?.objecter ms_handle_connect 0x7f4e040b1730 2024-03-25T18:23:59.814 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.812+0000 7f4e00ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:59.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.818+0000 7f4e00ff9640 10 client.?.objecter ms_handle_connect 0x7f4e040b4d60 2024-03-25T18:23:59.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.818+0000 7f4e00ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:59.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.822+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098677878 0 0) 0x7f4e0413a0a0 con 0x7f4e0414be90 2024-03-25T18:23:59.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.822+0000 7f4e00ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:59.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.822+0000 7f4e00ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:59.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.822+0000 7f4e00ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:59.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.822+0000 7f4e00ff9640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:23:59.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.822+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df0004eb0 con 0x7f4e0414be90 2024-03-25T18:23:59.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.825+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1802164055 0 0) 0x7f4e04134f90 con 0x7f4e040b1730 2024-03-25T18:23:59.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.825+0000 7f4e00ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:59.826 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.825+0000 7f4e00ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:59.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.825+0000 7f4e00ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:59.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.825+0000 7f4e00ff9640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:23:59.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.825+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0413a0a0 con 0x7f4e040b1730 2024-03-25T18:23:59.836 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.835+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3502260104 0 0) 0x7f4e0413a0a0 con 0x7f4e040b1730 2024-03-25T18:23:59.836 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.835+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e04134f90 con 0x7f4e040b1730 2024-03-25T18:23:59.836 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.835+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4df4001b00 con 0x7f4e040b1730 2024-03-25T18:23:59.836 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.835+0000 7f4e00ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:59.836 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.835+0000 7f4e00ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:59.836 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.835+0000 7f4e00ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:59.836 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:23:59.837 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:23:59.837 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:59.837 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:23:59.837 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:59.837 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:23:59.837 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:23:59.837 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:23:59.837 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:23:59.837 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:23:59.837 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.835+0000 7f4e00ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:59.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.840+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212714828 0 0) 0x7f4e04142d20 con 0x7f4e040b4d60 2024-03-25T18:23:59.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.840+0000 7f4e00ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:59.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.840+0000 7f4e00ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:59.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.840+0000 7f4e00ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:59.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.840+0000 7f4e00ff9640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:23:59.841 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.840+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0413a0a0 con 0x7f4e040b4d60 2024-03-25T18:23:59.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3516328133 0 0) 0x7f4e04134f90 con 0x7f4e040b1730 2024-03-25T18:23:59.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 >> v1:172.21.15.122:6790/0 conn(0x7f4e0414be90 legacy=0x7f4e04134880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:59.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 >> v1:172.21.15.122:6789/0 conn(0x7f4e040b4d60 legacy=0x7f4e04134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:59.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e00ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:23:59.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e00ff9640 1 monclient: found mon.b 2024-03-25T18:23:59.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e00ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:59.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e00ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:59.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e04140a90 con 0x7f4e040b1730 2024-03-25T18:23:59.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e00ff9640 10 monclient: _finish_auth 0 2024-03-25T18:23:59.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e00ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:23:59.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e00ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:59.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e0a838640 5 monclient: authenticate success, global_id 4564 2024-03-25T18:23:59.845 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e0a838640 10 monclient: _renew_subs 2024-03-25T18:23:59.845 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e0a838640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:59.845 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e0a838640 1 -- 172.21.15.122:0/652970553 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e04139190 con 0x7f4e040b1730 2024-03-25T18:23:59.845 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e0a838640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:59.845 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e0a838640 10 monclient: _renew_subs 2024-03-25T18:23:59.845 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e0a838640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:59.845 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e0a838640 1 -- 172.21.15.122:0/652970553 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e04143390 con 0x7f4e040b1730 2024-03-25T18:23:59.845 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.842+0000 7f4e0a838640 1 librados: init done 2024-03-25T18:23:59.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.843+0000 7f4e0a838640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:59.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.843+0000 7f4e0a838640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:59.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.843+0000 7f4e0a838640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:23:59.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:23:59.843+0000 7f4e0a838640 1 -- 172.21.15.122:0/652970553 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e040afbd0 con 0x7f4e040b1730 2024-03-25T18:24:00.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.119+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4df4001ca0 con 0x7f4e040b1730 2024-03-25T18:24:00.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.119+0000 7f4e00ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:00.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.119+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f4df4003a40 con 0x7f4e040b1730 2024-03-25T18:24:00.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.119+0000 7f4e00ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:00.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.119+0000 7f4e00ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:00.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.119+0000 7f4e00ff9640 10 monclient: dump: 2024-03-25T18:24:00.121 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:00.121 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:00.121 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:00.121 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:00.121 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:00.121 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:00.121 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:00.122 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:00.122 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:00.122 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:00.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.119+0000 7f4e027fc640 4 set_mon_vals no callback set 2024-03-25T18:24:00.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.120+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f4df4019a10 con 0x7f4e040b1730 2024-03-25T18:24:00.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.120+0000 7f4e037fe640 10 monclient: get_auth_request con 0x7f4df006ddc0 auth_method 0 2024-03-25T18:24:00.123 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.121+0000 7f4e00ff9640 10 client.4564.objecter ms_handle_connect 0x7f4df006ddc0 2024-03-25T18:24:00.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.126+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f4df40535a0 con 0x7f4e040b1730 2024-03-25T18:24:00.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.126+0000 7f4e00ff9640 10 client.4564.objecter ms_dispatch 0x7f4e04000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:00.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.126+0000 7f4e00ff9640 3 client.4564.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:00.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.126+0000 7f4e00ff9640 3 client.4564.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:00.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.126+0000 7f4e00ff9640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-25T18:24:00.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.126+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4df404eee0 con 0x7f4e040b1730 2024-03-25T18:24:00.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.126+0000 7f4e00ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:00.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.126+0000 7f4e00ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:00.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.379+0000 7f4e0a838640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:00.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.379+0000 7f4e0a838640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:00.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.379+0000 7f4e0a838640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:00.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.379+0000 7f4e0a838640 1 -- 172.21.15.122:0/652970553 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e040627b0 con 0x7f4e040b1730 2024-03-25T18:24:00.400 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.398+0000 7f4e00ff9640 1 -- 172.21.15.122:0/652970553 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f4df4050fe0 con 0x7f4e040b1730 2024-03-25T18:24:00.400 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.399+0000 7f4e00ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:00.400 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.399+0000 7f4e00ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:00.400 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:24:00.400 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:00.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.402+0000 7f4e0a838640 10 librados: watch_flush enter 2024-03-25T18:24:00.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.402+0000 7f4e0a838640 10 librados: watch_flush exit 2024-03-25T18:24:00.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.402+0000 7f4e0a838640 20 client.4564.objecter shutdown clearing up homeless session... 2024-03-25T18:24:00.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.402+0000 7f4e0a838640 10 client.4564.objecter successfully canceled tick 2024-03-25T18:24:00.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.402+0000 7f4e0a838640 1 -- 172.21.15.122:0/652970553 >> v1:172.21.15.204:6800/676228005 conn(0x7f4df006ddc0 legacy=0x7f4df0070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:00.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.402+0000 7f4e0a838640 10 monclient: shutdown 2024-03-25T18:24:00.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.402+0000 7f4e0a838640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:00.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.402+0000 7f4e0a838640 1 -- 172.21.15.122:0/652970553 >> v1:172.21.15.204:6789/0 conn(0x7f4e040b1730 legacy=0x7f4e04140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:00.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.402+0000 7f4e0a838640 1 -- 172.21.15.122:0/652970553 shutdown_connections 2024-03-25T18:24:00.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.402+0000 7f4e0a838640 1 -- 172.21.15.122:0/652970553 >> 172.21.15.122:0/652970553 conn(0x7f4e040617a0 msgr2=0x7f4e040b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:00.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.402+0000 7f4e0a838640 1 -- 172.21.15.122:0/652970553 shutdown_connections 2024-03-25T18:24:00.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.402+0000 7f4e0a838640 1 -- 172.21.15.122:0/652970553 wait complete. 2024-03-25T18:24:00.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:00.402+0000 7f4e0a838640 1 librados: shutdown 2024-03-25T18:24:00.415 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:07.220 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:07.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.333+0000 7ff80a064640 10 monclient: get_monmap_and_config 2024-03-25T18:24:07.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.333+0000 7ff80a064640 10 monclient: build_initial_monmap 2024-03-25T18:24:07.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.333+0000 7ff80a064640 10 monclient: monmap: 2024-03-25T18:24:07.334 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:24:07.334 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:07.334 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:24:07.334453+0000 2024-03-25T18:24:07.335 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:24:07.334453+0000 2024-03-25T18:24:07.335 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:07.335 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:07.335 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:24:07.335 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:24:07.335 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:24:07.335 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:07.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.334+0000 7ff80a064640 1 Processor -- start 2024-03-25T18:24:07.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.334+0000 7ff80a064640 1 -- start start 2024-03-25T18:24:07.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.334+0000 7ff80a064640 10 monclient: init 2024-03-25T18:24:07.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.334+0000 7ff80a064640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:07.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.334+0000 7ff80a064640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:07.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.334+0000 7ff80a064640 10 monclient(hunting): picked mon.noname-a con 0x7ff804061830 addr v1:172.21.15.122:6789/0 2024-03-25T18:24:07.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.334+0000 7ff80a064640 10 monclient(hunting): picked mon.noname-b con 0x7ff80413d9f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:24:07.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.334+0000 7ff80a064640 10 monclient(hunting): picked mon.noname-c con 0x7ff804143d50 addr v1:172.21.15.204:6789/0 2024-03-25T18:24:07.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.334+0000 7ff80a064640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff804071af0 con 0x7ff804061830 2024-03-25T18:24:07.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.334+0000 7ff80a064640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8041441c0 con 0x7ff804143d50 2024-03-25T18:24:07.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.334+0000 7ff80a064640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff804072810 con 0x7ff80413d9f0 2024-03-25T18:24:07.336 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.334+0000 7ff80a064640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:07.337 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.334+0000 7ff80a064640 10 monclient(hunting): authenticate will time out at 7770.600828s 2024-03-25T18:24:07.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.340+0000 7ff803fff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7ff804143d50 0x7ff80414b9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:59138/0 (socket says 172.21.15.122:59138) 2024-03-25T18:24:07.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.340+0000 7ff803fff640 1 -- 172.21.15.122:0/1007768419 learned_addr learned my addr 172.21.15.122:0/1007768419 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:24:07.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.348+0000 7ff8027fc640 1 -- 172.21.15.122:0/1007768419 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 624163711 0 0) 0x7ff8041441c0 con 0x7ff804143d50 2024-03-25T18:24:07.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.348+0000 7ff8027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:07.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.348+0000 7ff8027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:07.349 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.348+0000 7ff8027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:07.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.348+0000 7ff8027fc640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:24:07.350 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.348+0000 7ff8027fc640 1 -- 172.21.15.122:0/1007768419 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7ec003520 con 0x7ff804143d50 2024-03-25T18:24:07.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.352+0000 7ff8027fc640 1 -- 172.21.15.122:0/1007768419 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601357083 0 0) 0x7ff804071af0 con 0x7ff804061830 2024-03-25T18:24:07.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.352+0000 7ff8027fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:07.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.352+0000 7ff8027fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:07.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.352+0000 7ff8027fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:07.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.352+0000 7ff8027fc640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:24:07.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.352+0000 7ff8027fc640 1 -- 172.21.15.122:0/1007768419 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8041441c0 con 0x7ff804061830 2024-03-25T18:24:07.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.354+0000 7ff8027fc640 1 -- 172.21.15.122:0/1007768419 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3303159678 0 0) 0x7ff7ec003520 con 0x7ff804143d50 2024-03-25T18:24:07.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.354+0000 7ff8027fc640 1 -- 172.21.15.122:0/1007768419 >> v1:172.21.15.122:6790/0 conn(0x7ff80413d9f0 legacy=0x7ff804143640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:24:07.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.354+0000 7ff8027fc640 1 -- 172.21.15.122:0/1007768419 >> v1:172.21.15.122:6789/0 conn(0x7ff804061830 legacy=0x7ff80413d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:07.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.354+0000 7ff8027fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:07.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.354+0000 7ff8027fc640 1 monclient: found mon.noname-c 2024-03-25T18:24:07.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.354+0000 7ff8027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:07.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.354+0000 7ff8027fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:24:07.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.354+0000 7ff8027fc640 1 -- 172.21.15.122:0/1007768419 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff804072af0 con 0x7ff804143d50 2024-03-25T18:24:07.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.354+0000 7ff8027fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:07.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.354+0000 7ff8027fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:07.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.354+0000 7ff8027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:07.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.354+0000 7ff8027fc640 1 -- 172.21.15.122:0/1007768419 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff7fc001a10 con 0x7ff804143d50 2024-03-25T18:24:07.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.354+0000 7ff8027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:07.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.354+0000 7ff8027fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:07.356 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.354+0000 7ff8027fc640 10 monclient: dump: 2024-03-25T18:24:07.356 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:07.356 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:07.356 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:07.356 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:07.356 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:07.356 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:07.356 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:07.356 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:07.356 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:07.356 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:07.357 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.354+0000 7ff80a064640 5 monclient: authenticate success, global_id 4567 2024-03-25T18:24:07.357 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.354+0000 7ff80a064640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:07.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.359+0000 7ff8027fc640 1 -- 172.21.15.122:0/1007768419 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7fc001b70 con 0x7ff804143d50 2024-03-25T18:24:07.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.359+0000 7ff8027fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:07.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.359+0000 7ff80a064640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:07.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.359+0000 7ff80a064640 4 set_mon_vals no callback set 2024-03-25T18:24:07.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.360+0000 7ff8027fc640 1 -- 172.21.15.122:0/1007768419 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff7fc001f80 con 0x7ff804143d50 2024-03-25T18:24:07.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.360+0000 7ff8027fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:07.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.360+0000 7ff8027fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:07.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.360+0000 7ff8027fc640 10 monclient: dump: 2024-03-25T18:24:07.361 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:07.361 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:07.361 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:07.361 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:07.361 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:07.362 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:07.362 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:07.362 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:07.362 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:07.362 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:07.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.360+0000 7ff80a064640 10 monclient: shutdown 2024-03-25T18:24:07.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.360+0000 7ff80a064640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:07.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.360+0000 7ff80a064640 1 -- 172.21.15.122:0/1007768419 >> v1:172.21.15.204:6789/0 conn(0x7ff804143d50 legacy=0x7ff80414b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:07.364 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.360+0000 7ff80a064640 1 -- 172.21.15.122:0/1007768419 shutdown_connections 2024-03-25T18:24:07.364 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.360+0000 7ff80a064640 1 -- 172.21.15.122:0/1007768419 >> 172.21.15.122:0/1007768419 conn(0x7ff8040b11e0 msgr2=0x7ff8040b15f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:07.364 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.360+0000 7ff80a064640 1 -- 172.21.15.122:0/1007768419 shutdown_connections 2024-03-25T18:24:07.364 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.360+0000 7ff80a064640 1 -- 172.21.15.122:0/1007768419 wait complete. 2024-03-25T18:24:07.364 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.361+0000 7ff80a064640 10 monclient: build_initial_monmap 2024-03-25T18:24:07.364 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.361+0000 7ff80a064640 10 monclient: monmap: 2024-03-25T18:24:07.364 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:24:07.364 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:07.364 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:24:07.364 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:24:07.364 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:07.364 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:24:07.364 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:24:07.364 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:24:07.364 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:24:07.365 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:07.365 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.362+0000 7ff80a064640 1 librados: starting msgr at 2024-03-25T18:24:07.365 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.362+0000 7ff80a064640 1 librados: starting objecter 2024-03-25T18:24:07.365 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.362+0000 7ff80a064640 1 Processor -- start 2024-03-25T18:24:07.365 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.362+0000 7ff80a064640 1 -- start start 2024-03-25T18:24:07.365 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.362+0000 7ff80a064640 1 librados: setting wanted keys 2024-03-25T18:24:07.365 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.362+0000 7ff80a064640 1 librados: calling monclient init 2024-03-25T18:24:07.365 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.362+0000 7ff80a064640 10 monclient: init 2024-03-25T18:24:07.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.363+0000 7ff80a064640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:07.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.363+0000 7ff80a064640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:07.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.363+0000 7ff80a064640 10 monclient(hunting): picked mon.noname-a con 0x7ff804061830 addr v1:172.21.15.122:6789/0 2024-03-25T18:24:07.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.363+0000 7ff80a064640 10 monclient(hunting): picked mon.noname-c con 0x7ff80413d9f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:24:07.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.363+0000 7ff80a064640 10 monclient(hunting): picked mon.noname-b con 0x7ff804143d50 addr v1:172.21.15.204:6789/0 2024-03-25T18:24:07.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.363+0000 7ff80a064640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff804140a50 con 0x7ff804061830 2024-03-25T18:24:07.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.363+0000 7ff80a064640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80413f570 con 0x7ff804143d50 2024-03-25T18:24:07.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.363+0000 7ff80a064640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff80413ef50 con 0x7ff80413d9f0 2024-03-25T18:24:07.367 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.363+0000 7ff80a064640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:07.367 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.363+0000 7ff80a064640 10 monclient(hunting): authenticate will time out at 7770.629278s 2024-03-25T18:24:07.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.373+0000 7ff803fff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7ff804143d50 0x7ff8041ddbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:59144/0 (socket says 172.21.15.122:59144) 2024-03-25T18:24:07.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.373+0000 7ff803fff640 1 -- 172.21.15.122:0/292114365 learned_addr learned my addr 172.21.15.122:0/292114365 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:24:07.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.373+0000 7ff7ebfff640 10 client.?.objecter ms_handle_connect 0x7ff804143d50 2024-03-25T18:24:07.374 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.373+0000 7ff7ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:07.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.379+0000 7ff7ebfff640 10 client.?.objecter ms_handle_connect 0x7ff804061830 2024-03-25T18:24:07.380 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.379+0000 7ff7ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:07.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.386+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762490417 0 0) 0x7ff80413f570 con 0x7ff804143d50 2024-03-25T18:24:07.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.386+0000 7ff7ebfff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:07.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.386+0000 7ff7ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:07.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.386+0000 7ff7ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:07.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.386+0000 7ff7ebfff640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:24:07.388 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.386+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7f0004eb0 con 0x7ff804143d50 2024-03-25T18:24:07.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.388+0000 7ff7ebfff640 10 client.?.objecter ms_handle_connect 0x7ff80413d9f0 2024-03-25T18:24:07.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.388+0000 7ff7ebfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:07.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.397+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740235934 0 0) 0x7ff804140a50 con 0x7ff804061830 2024-03-25T18:24:07.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.397+0000 7ff7ebfff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:07.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.397+0000 7ff7ebfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:07.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.397+0000 7ff7ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:07.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.397+0000 7ff7ebfff640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:24:07.399 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.397+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff80413f570 con 0x7ff804061830 2024-03-25T18:24:07.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.402+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 99210696 0 0) 0x7ff7f0004eb0 con 0x7ff804143d50 2024-03-25T18:24:07.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.402+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff804140a50 con 0x7ff804143d50 2024-03-25T18:24:07.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.402+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff7fc001f40 con 0x7ff804143d50 2024-03-25T18:24:07.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.402+0000 7ff7ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:07.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.402+0000 7ff7ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:07.404 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.402+0000 7ff7ebfff640 10 monclient(hunting): dump: 2024-03-25T18:24:07.404 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:07.404 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:07.404 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:07.404 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:07.404 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:07.404 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:07.404 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:07.405 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:07.405 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:07.405 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:07.405 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.403+0000 7ff7ebfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:07.412 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.411+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2455630896 0 0) 0x7ff80413f570 con 0x7ff804061830 2024-03-25T18:24:07.412 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.411+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff7f0004eb0 con 0x7ff804061830 2024-03-25T18:24:07.412 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.411+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff7f40018b0 con 0x7ff804061830 2024-03-25T18:24:07.412 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.411+0000 7ff7ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:07.412 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.411+0000 7ff7ebfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:07.413 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.411+0000 7ff7ebfff640 10 monclient(hunting): dump: 2024-03-25T18:24:07.413 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:07.413 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:07.413 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:07.413 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:07.413 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:07.413 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:07.413 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:07.413 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:07.413 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:07.413 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:07.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2391890275 0 0) 0x7ff804140a50 con 0x7ff804143d50 2024-03-25T18:24:07.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 >> v1:172.21.15.122:6790/0 conn(0x7ff80413d9f0 legacy=0x7ff8040ab660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:07.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 >> v1:172.21.15.122:6789/0 conn(0x7ff804061830 legacy=0x7ff80413e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:07.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff7ebfff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:07.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff7ebfff640 1 monclient: found mon.b 2024-03-25T18:24:07.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff7ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:07.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff7ebfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:07.415 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80413dec0 con 0x7ff804143d50 2024-03-25T18:24:07.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff7ebfff640 10 monclient: _finish_auth 0 2024-03-25T18:24:07.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff7ebfff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:07.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff7ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:07.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff80a064640 5 monclient: authenticate success, global_id 4570 2024-03-25T18:24:07.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff80a064640 10 monclient: _renew_subs 2024-03-25T18:24:07.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff80a064640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:07.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff80a064640 1 -- 172.21.15.122:0/292114365 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff80413f170 con 0x7ff804143d50 2024-03-25T18:24:07.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff80a064640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:07.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff80a064640 10 monclient: _renew_subs 2024-03-25T18:24:07.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff80a064640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:07.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff80a064640 1 -- 172.21.15.122:0/292114365 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8040abd70 con 0x7ff804143d50 2024-03-25T18:24:07.416 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.413+0000 7ff80a064640 1 librados: init done 2024-03-25T18:24:07.417 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.414+0000 7ff80a064640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:07.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.414+0000 7ff80a064640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:07.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.414+0000 7ff80a064640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:07.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.414+0000 7ff80a064640 1 -- 172.21.15.122:0/292114365 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8040ac020 con 0x7ff804143d50 2024-03-25T18:24:07.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.690+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7fc004e10 con 0x7ff804143d50 2024-03-25T18:24:07.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.690+0000 7ff7ebfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:07.691 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.690+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff7fc005120 con 0x7ff804143d50 2024-03-25T18:24:07.693 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.690+0000 7ff7ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:07.693 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.690+0000 7ff7ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:07.693 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.690+0000 7ff7ebfff640 10 monclient: dump: 2024-03-25T18:24:07.693 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:07.693 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:07.693 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:07.693 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:07.693 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:07.694 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:07.694 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:07.694 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:07.694 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:07.694 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:07.694 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.690+0000 7ff801ffb640 4 set_mon_vals no callback set 2024-03-25T18:24:07.694 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.691+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7ff7fc01b0f0 con 0x7ff804143d50 2024-03-25T18:24:07.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.691+0000 7ff8037fe640 10 monclient: get_auth_request con 0x7ff7f006ded0 auth_method 0 2024-03-25T18:24:07.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.692+0000 7ff7ebfff640 10 client.4570.objecter ms_handle_connect 0x7ff7f006ded0 2024-03-25T18:24:07.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.696+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7ff7fc054c20 con 0x7ff804143d50 2024-03-25T18:24:07.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.696+0000 7ff7ebfff640 10 client.4570.objecter ms_dispatch 0x7ff804000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:07.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.696+0000 7ff7ebfff640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:07.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.696+0000 7ff7ebfff640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:07.698 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.696+0000 7ff7ebfff640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-25T18:24:07.698 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.697+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff7fc050560 con 0x7ff804143d50 2024-03-25T18:24:07.698 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.697+0000 7ff7ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:07.698 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.697+0000 7ff7ebfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:07.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.947+0000 7ff80a064640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:07.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.947+0000 7ff80a064640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:07.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.947+0000 7ff80a064640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:07.948 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.947+0000 7ff80a064640 1 -- 172.21.15.122:0/292114365 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff80413f300 con 0x7ff804143d50 2024-03-25T18:24:07.966 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.965+0000 7ff7ebfff640 1 -- 172.21.15.122:0/292114365 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7ff7fc052660 con 0x7ff804143d50 2024-03-25T18:24:07.966 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.965+0000 7ff7ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:07.966 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.965+0000 7ff7ebfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:07.967 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:24:07.967 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:07.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.967+0000 7ff80a064640 10 librados: watch_flush enter 2024-03-25T18:24:07.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.967+0000 7ff80a064640 10 librados: watch_flush exit 2024-03-25T18:24:07.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.967+0000 7ff80a064640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-25T18:24:07.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.967+0000 7ff80a064640 10 client.4570.objecter successfully canceled tick 2024-03-25T18:24:07.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.968+0000 7ff80a064640 1 -- 172.21.15.122:0/292114365 >> v1:172.21.15.204:6800/676228005 conn(0x7ff7f006ded0 legacy=0x7ff7f0070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:07.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.968+0000 7ff80a064640 10 monclient: shutdown 2024-03-25T18:24:07.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.968+0000 7ff80a064640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:07.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.968+0000 7ff80a064640 1 -- 172.21.15.122:0/292114365 >> v1:172.21.15.204:6789/0 conn(0x7ff804143d50 legacy=0x7ff8041ddbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:07.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.968+0000 7ff80a064640 1 -- 172.21.15.122:0/292114365 shutdown_connections 2024-03-25T18:24:07.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.968+0000 7ff80a064640 1 -- 172.21.15.122:0/292114365 >> 172.21.15.122:0/292114365 conn(0x7ff8040b11e0 msgr2=0x7ff80413bd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:07.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.968+0000 7ff80a064640 1 -- 172.21.15.122:0/292114365 shutdown_connections 2024-03-25T18:24:07.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.968+0000 7ff80a064640 1 -- 172.21.15.122:0/292114365 wait complete. 2024-03-25T18:24:07.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:07.968+0000 7ff80a064640 1 librados: shutdown 2024-03-25T18:24:07.981 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:12.063 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:08:05.805 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:12.086 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:08:05.828 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:12.121 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:08:05.864 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:12.128 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:08:05.871 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:12.143 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:08:05.885 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:12.152 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:08:05.897 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:12.158 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:08:05.903 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:12.166 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:08:05.908 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:12.182 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:08:05.928 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:14.788 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:14.888 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.887+0000 7f753177a640 10 monclient: get_monmap_and_config 2024-03-25T18:24:14.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.887+0000 7f753177a640 10 monclient: build_initial_monmap 2024-03-25T18:24:14.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.887+0000 7f753177a640 10 monclient: monmap: 2024-03-25T18:24:14.889 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:24:14.889 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:14.889 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:24:14.888869+0000 2024-03-25T18:24:14.889 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:24:14.888869+0000 2024-03-25T18:24:14.889 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:14.889 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:14.889 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:24:14.889 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:24:14.889 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:24:14.889 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:14.890 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.888+0000 7f753177a640 1 Processor -- start 2024-03-25T18:24:14.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.888+0000 7f753177a640 1 -- start start 2024-03-25T18:24:14.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.888+0000 7f753177a640 10 monclient: init 2024-03-25T18:24:14.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.888+0000 7f753177a640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:14.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.888+0000 7f753177a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:14.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.888+0000 7f753177a640 10 monclient(hunting): picked mon.noname-a con 0x7f752c0a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:24:14.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.888+0000 7f753177a640 10 monclient(hunting): picked mon.noname-b con 0x7f752c13f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:24:14.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.888+0000 7f753177a640 10 monclient(hunting): picked mon.noname-c con 0x7f752c1430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:24:14.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.888+0000 7f753177a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c142520 con 0x7f752c0a98f0 2024-03-25T18:24:14.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.888+0000 7f753177a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c0b28a0 con 0x7f752c1430c0 2024-03-25T18:24:14.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.889+0000 7f753177a640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c0b2a30 con 0x7f752c13f710 2024-03-25T18:24:14.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.889+0000 7f753177a640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:14.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.889+0000 7f753177a640 10 monclient(hunting): authenticate will time out at 7778.154989s 2024-03-25T18:24:14.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.889+0000 7f752b7fe640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f752c1430c0 0x7f752c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:49722/0 (socket says 172.21.15.122:49722) 2024-03-25T18:24:14.891 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.889+0000 7f752b7fe640 1 -- 172.21.15.122:0/1906208234 learned_addr learned my addr 172.21.15.122:0/1906208234 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:24:14.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.966+0000 7f7529ffb640 1 -- 172.21.15.122:0/1906208234 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392559506 0 0) 0x7f752c142520 con 0x7f752c0a98f0 2024-03-25T18:24:14.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.966+0000 7f7529ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:14.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.966+0000 7f7529ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:14.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.966+0000 7f7529ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:14.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.966+0000 7f7529ffb640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:24:14.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.966+0000 7f7529ffb640 1 -- 172.21.15.122:0/1906208234 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7514003520 con 0x7f752c0a98f0 2024-03-25T18:24:14.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.968+0000 7f7529ffb640 1 -- 172.21.15.122:0/1906208234 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440803012 0 0) 0x7f752c0b28a0 con 0x7f752c1430c0 2024-03-25T18:24:14.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.968+0000 7f7529ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:14.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.968+0000 7f7529ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:14.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.968+0000 7f7529ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:14.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.968+0000 7f7529ffb640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:24:14.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.968+0000 7f7529ffb640 1 -- 172.21.15.122:0/1906208234 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c142520 con 0x7f752c1430c0 2024-03-25T18:24:14.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.978+0000 7f7529ffb640 1 -- 172.21.15.122:0/1906208234 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2251088846 0 0) 0x7f752c0b2a30 con 0x7f752c13f710 2024-03-25T18:24:14.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.978+0000 7f7529ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:24:14.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.978+0000 7f7529ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:14.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.978+0000 7f7529ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:14.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.978+0000 7f7529ffb640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:24:14.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.978+0000 7f7529ffb640 1 -- 172.21.15.122:0/1906208234 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c0b28a0 con 0x7f752c13f710 2024-03-25T18:24:14.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f7529ffb640 1 -- 172.21.15.122:0/1906208234 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 191659368 0 0) 0x7f7514003520 con 0x7f752c0a98f0 2024-03-25T18:24:14.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f7529ffb640 1 -- 172.21.15.122:0/1906208234 >> v1:172.21.15.122:6790/0 conn(0x7f752c13f710 legacy=0x7f752c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:14.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f7529ffb640 1 -- 172.21.15.122:0/1906208234 >> v1:172.21.15.204:6789/0 conn(0x7f752c1430c0 legacy=0x7f752c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:14.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f7529ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:24:14.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f7529ffb640 1 monclient: found mon.noname-a 2024-03-25T18:24:14.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f7529ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:14.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f7529ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:24:14.985 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f7529ffb640 1 -- 172.21.15.122:0/1906208234 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c0b44c0 con 0x7f752c0a98f0 2024-03-25T18:24:14.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f7529ffb640 10 monclient: _finish_auth 0 2024-03-25T18:24:14.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f7529ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:24:14.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f7529ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:14.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f7529ffb640 1 -- 172.21.15.122:0/1906208234 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f751c001690 con 0x7f752c0a98f0 2024-03-25T18:24:14.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f753177a640 5 monclient: authenticate success, global_id 4581 2024-03-25T18:24:14.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f753177a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:14.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f7529ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:14.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f7529ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:14.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f7529ffb640 10 monclient: dump: 2024-03-25T18:24:14.986 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:14.986 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:14.986 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:14.986 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:14.986 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:14.986 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:14.987 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:14.987 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:14.987 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:14.987 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:14.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:14.983+0000 7f753177a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:15.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.004+0000 7f7529ffb640 1 -- 172.21.15.122:0/1906208234 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f751c0017f0 con 0x7f752c0a98f0 2024-03-25T18:24:15.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.004+0000 7f7529ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:15.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.004+0000 7f7529ffb640 1 -- 172.21.15.122:0/1906208234 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f751c001c00 con 0x7f752c0a98f0 2024-03-25T18:24:15.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.004+0000 7f753177a640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:15.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.004+0000 7f753177a640 4 set_mon_vals no callback set 2024-03-25T18:24:15.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.004+0000 7f7529ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:15.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.004+0000 7f7529ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:15.006 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.004+0000 7f7529ffb640 10 monclient: dump: 2024-03-25T18:24:15.006 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:15.006 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:15.006 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:15.006 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:15.006 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:15.006 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:15.006 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:15.006 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:15.006 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:15.006 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:15.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.004+0000 7f753177a640 10 monclient: shutdown 2024-03-25T18:24:15.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.004+0000 7f753177a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:15.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.004+0000 7f753177a640 1 -- 172.21.15.122:0/1906208234 >> v1:172.21.15.122:6789/0 conn(0x7f752c0a98f0 legacy=0x7f752c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:15.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.004+0000 7f753177a640 1 -- 172.21.15.122:0/1906208234 shutdown_connections 2024-03-25T18:24:15.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.004+0000 7f753177a640 1 -- 172.21.15.122:0/1906208234 >> 172.21.15.122:0/1906208234 conn(0x7f752c0b7ea0 msgr2=0x7f752c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:15.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.005+0000 7f753177a640 1 -- 172.21.15.122:0/1906208234 shutdown_connections 2024-03-25T18:24:15.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.005+0000 7f753177a640 1 -- 172.21.15.122:0/1906208234 wait complete. 2024-03-25T18:24:15.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.005+0000 7f753177a640 10 monclient: build_initial_monmap 2024-03-25T18:24:15.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.006+0000 7f753177a640 10 monclient: monmap: 2024-03-25T18:24:15.008 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:24:15.008 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:15.008 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:24:15.008 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:24:15.008 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:15.009 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:24:15.009 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:24:15.009 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:24:15.009 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:24:15.009 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:15.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.006+0000 7f753177a640 1 librados: starting msgr at 2024-03-25T18:24:15.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.006+0000 7f753177a640 1 librados: starting objecter 2024-03-25T18:24:15.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.006+0000 7f753177a640 1 Processor -- start 2024-03-25T18:24:15.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.006+0000 7f753177a640 1 -- start start 2024-03-25T18:24:15.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.006+0000 7f753177a640 1 librados: setting wanted keys 2024-03-25T18:24:15.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.006+0000 7f753177a640 1 librados: calling monclient init 2024-03-25T18:24:15.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.006+0000 7f753177a640 10 monclient: init 2024-03-25T18:24:15.010 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.007+0000 7f753177a640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:15.010 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.007+0000 7f753177a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:15.010 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.007+0000 7f753177a640 10 monclient(hunting): picked mon.noname-b con 0x7f752c0a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:24:15.010 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.007+0000 7f753177a640 10 monclient(hunting): picked mon.noname-c con 0x7f752c13f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:24:15.010 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.007+0000 7f753177a640 10 monclient(hunting): picked mon.noname-a con 0x7f752c1430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:24:15.010 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.007+0000 7f753177a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c1db050 con 0x7f752c1430c0 2024-03-25T18:24:15.010 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.007+0000 7f753177a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c1dab70 con 0x7f752c0a98f0 2024-03-25T18:24:15.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.007+0000 7f753177a640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f752c1dad00 con 0x7f752c13f710 2024-03-25T18:24:15.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.007+0000 7f753177a640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:15.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.007+0000 7f753177a640 10 monclient(hunting): authenticate will time out at 7778.273633s 2024-03-25T18:24:15.019 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.018+0000 7f752b7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f752c1430c0 0x7f752c1393b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:36790/0 (socket says 172.21.15.122:36790) 2024-03-25T18:24:15.019 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.018+0000 7f752b7fe640 1 -- 172.21.15.122:0/1144948178 learned_addr learned my addr 172.21.15.122:0/1144948178 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:24:15.023 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.022+0000 7f75137fe640 10 client.?.objecter ms_handle_connect 0x7f752c0a98f0 2024-03-25T18:24:15.023 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.022+0000 7f75137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:15.025 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.024+0000 7f75137fe640 10 client.?.objecter ms_handle_connect 0x7f752c1430c0 2024-03-25T18:24:15.025 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.024+0000 7f75137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:15.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.037+0000 7f75137fe640 10 client.?.objecter ms_handle_connect 0x7f752c13f710 2024-03-25T18:24:15.038 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.037+0000 7f75137fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:15.040 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.039+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1807481904 0 0) 0x7f752c1db050 con 0x7f752c1430c0 2024-03-25T18:24:15.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.039+0000 7f75137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:15.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.039+0000 7f75137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:15.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.039+0000 7f75137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:15.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.039+0000 7f75137fe640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:24:15.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.039+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7518004eb0 con 0x7f752c1430c0 2024-03-25T18:24:15.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.040+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655707770 0 0) 0x7f752c1dab70 con 0x7f752c0a98f0 2024-03-25T18:24:15.041 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.040+0000 7f75137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:15.042 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.040+0000 7f75137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:15.042 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.040+0000 7f75137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:15.042 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.040+0000 7f75137fe640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:24:15.042 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.040+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c1db050 con 0x7f752c0a98f0 2024-03-25T18:24:15.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.053+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1322961783 0 0) 0x7f752c1dad00 con 0x7f752c13f710 2024-03-25T18:24:15.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.053+0000 7f75137fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:15.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.053+0000 7f75137fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:15.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.053+0000 7f75137fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:15.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.053+0000 7f75137fe640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:24:15.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.053+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f752c1dab70 con 0x7f752c13f710 2024-03-25T18:24:15.062 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.061+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 91319892 0 0) 0x7f752c1db050 con 0x7f752c0a98f0 2024-03-25T18:24:15.062 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.061+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f752c1dad00 con 0x7f752c0a98f0 2024-03-25T18:24:15.063 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.061+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f751c001b00 con 0x7f752c0a98f0 2024-03-25T18:24:15.063 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.061+0000 7f75137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:15.063 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.061+0000 7f75137fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:15.063 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.061+0000 7f75137fe640 10 monclient(hunting): dump: 2024-03-25T18:24:15.063 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:15.063 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:15.063 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:15.063 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:15.063 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:15.063 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:15.063 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:15.063 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:15.063 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:15.063 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:15.063 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.061+0000 7f75137fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:15.064 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.063+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4188449792 0 0) 0x7f7518004eb0 con 0x7f752c1430c0 2024-03-25T18:24:15.064 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.063+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f752c1db050 con 0x7f752c1430c0 2024-03-25T18:24:15.064 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.063+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f7524001c70 con 0x7f752c1430c0 2024-03-25T18:24:15.064 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.063+0000 7f75137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:15.065 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.063+0000 7f75137fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:15.065 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.063+0000 7f75137fe640 10 monclient(hunting): dump: 2024-03-25T18:24:15.065 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:15.065 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:15.065 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:15.065 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:15.065 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:15.065 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:15.065 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:15.065 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:15.065 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:15.065 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:15.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.073+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 614663774 0 0) 0x7f752c1dab70 con 0x7f752c13f710 2024-03-25T18:24:15.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.073+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7518004eb0 con 0x7f752c13f710 2024-03-25T18:24:15.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.073+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f7520001830 con 0x7f752c13f710 2024-03-25T18:24:15.074 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.073+0000 7f75137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:15.075 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.073+0000 7f75137fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:15.075 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.073+0000 7f75137fe640 10 monclient(hunting): dump: 2024-03-25T18:24:15.075 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:15.075 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:15.075 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:15.075 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:15.075 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:15.075 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:15.075 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:15.075 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:15.075 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:15.075 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:15.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1689658720 0 0) 0x7f752c1dad00 con 0x7f752c0a98f0 2024-03-25T18:24:15.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 >> v1:172.21.15.122:6790/0 conn(0x7f752c13f710 legacy=0x7f752c1d7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:15.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 >> v1:172.21.15.122:6789/0 conn(0x7f752c1430c0 legacy=0x7f752c1393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:15.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f75137fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:15.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f75137fe640 1 monclient: found mon.b 2024-03-25T18:24:15.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f75137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:15.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f75137fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:15.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f752c0b70a0 con 0x7f752c0a98f0 2024-03-25T18:24:15.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f75137fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:15.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f75137fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:15.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f75137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:15.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f753177a640 5 monclient: authenticate success, global_id 4576 2024-03-25T18:24:15.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f753177a640 10 monclient: _renew_subs 2024-03-25T18:24:15.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f753177a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:15.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f753177a640 1 -- 172.21.15.122:0/1144948178 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f752c0b72f0 con 0x7f752c0a98f0 2024-03-25T18:24:15.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f753177a640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:15.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f753177a640 10 monclient: _renew_subs 2024-03-25T18:24:15.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f753177a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:15.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f753177a640 1 -- 172.21.15.122:0/1144948178 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f752c1e32a0 con 0x7f752c0a98f0 2024-03-25T18:24:15.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.083+0000 7f753177a640 1 librados: init done 2024-03-25T18:24:15.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.084+0000 7f753177a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:15.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.084+0000 7f753177a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:15.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.084+0000 7f753177a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:15.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.084+0000 7f753177a640 1 -- 172.21.15.122:0/1144948178 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f752c074cc0 con 0x7f752c0a98f0 2024-03-25T18:24:15.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.386+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f751c001ca0 con 0x7f752c0a98f0 2024-03-25T18:24:15.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.386+0000 7f75137fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:15.388 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.386+0000 7f75297fa640 4 set_mon_vals no callback set 2024-03-25T18:24:15.388 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.386+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f751c003a40 con 0x7f752c0a98f0 2024-03-25T18:24:15.388 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.387+0000 7f75137fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:15.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.387+0000 7f75137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:15.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.387+0000 7f75137fe640 10 monclient: dump: 2024-03-25T18:24:15.389 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:15.389 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:15.389 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:15.389 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:15.389 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:15.389 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:15.389 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:15.389 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:15.389 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:15.389 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:15.389 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.387+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f751c019a10 con 0x7f752c0a98f0 2024-03-25T18:24:15.390 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.388+0000 7f752a7fc640 10 monclient: get_auth_request con 0x7f751806e0f0 auth_method 0 2024-03-25T18:24:15.390 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.388+0000 7f75137fe640 10 client.4576.objecter ms_handle_connect 0x7f751806e0f0 2024-03-25T18:24:15.393 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.392+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f751c0535e0 con 0x7f752c0a98f0 2024-03-25T18:24:15.393 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.392+0000 7f75137fe640 10 client.4576.objecter ms_dispatch 0x7f752c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:15.393 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.392+0000 7f75137fe640 3 client.4576.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:15.393 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.392+0000 7f75137fe640 3 client.4576.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:15.393 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.392+0000 7f75137fe640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-25T18:24:15.393 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.392+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f751c04ef20 con 0x7f752c0a98f0 2024-03-25T18:24:15.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.392+0000 7f75137fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:15.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.392+0000 7f75137fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:15.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.651+0000 7f753177a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:15.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.651+0000 7f753177a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:15.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.651+0000 7f753177a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:15.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.651+0000 7f753177a640 1 -- 172.21.15.122:0/1144948178 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f752c0627b0 con 0x7f752c0a98f0 2024-03-25T18:24:15.679 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.678+0000 7f75137fe640 1 -- 172.21.15.122:0/1144948178 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f751c001210 con 0x7f752c0a98f0 2024-03-25T18:24:15.679 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.678+0000 7f75137fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:15.679 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.678+0000 7f75137fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:15.680 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:24:15.680 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:15.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.680+0000 7f753177a640 10 librados: watch_flush enter 2024-03-25T18:24:15.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.680+0000 7f753177a640 10 librados: watch_flush exit 2024-03-25T18:24:15.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.680+0000 7f753177a640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-25T18:24:15.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.680+0000 7f753177a640 10 client.4576.objecter successfully canceled tick 2024-03-25T18:24:15.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.680+0000 7f753177a640 1 -- 172.21.15.122:0/1144948178 >> v1:172.21.15.204:6800/676228005 conn(0x7f751806e0f0 legacy=0x7f75180705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:15.683 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.680+0000 7f753177a640 10 monclient: shutdown 2024-03-25T18:24:15.683 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.680+0000 7f753177a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:15.683 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.680+0000 7f753177a640 1 -- 172.21.15.122:0/1144948178 >> v1:172.21.15.204:6789/0 conn(0x7f752c0a98f0 legacy=0x7f752c1db4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:15.683 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.681+0000 7f753177a640 1 -- 172.21.15.122:0/1144948178 shutdown_connections 2024-03-25T18:24:15.683 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.681+0000 7f753177a640 1 -- 172.21.15.122:0/1144948178 >> 172.21.15.122:0/1144948178 conn(0x7f752c0b7ea0 msgr2=0x7f752c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:15.683 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.681+0000 7f753177a640 1 -- 172.21.15.122:0/1144948178 shutdown_connections 2024-03-25T18:24:15.683 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.681+0000 7f753177a640 1 -- 172.21.15.122:0/1144948178 wait complete. 2024-03-25T18:24:15.683 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:15.681+0000 7f753177a640 1 librados: shutdown 2024-03-25T18:24:15.693 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:22.500 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:22.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.615+0000 7f01afe4a640 10 monclient: get_monmap_and_config 2024-03-25T18:24:22.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.615+0000 7f01afe4a640 10 monclient: build_initial_monmap 2024-03-25T18:24:22.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.615+0000 7f01afe4a640 10 monclient: monmap: 2024-03-25T18:24:22.617 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:24:22.617 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:22.617 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:24:22.616494+0000 2024-03-25T18:24:22.617 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:24:22.616494+0000 2024-03-25T18:24:22.617 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:22.617 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:22.617 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:24:22.618 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:24:22.618 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:24:22.618 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:22.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.616+0000 7f01afe4a640 1 Processor -- start 2024-03-25T18:24:22.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.616+0000 7f01afe4a640 1 -- start start 2024-03-25T18:24:22.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.616+0000 7f01afe4a640 10 monclient: init 2024-03-25T18:24:22.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.617+0000 7f01afe4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:22.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.617+0000 7f01afe4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:22.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.617+0000 7f01afe4a640 10 monclient(hunting): picked mon.noname-c con 0x7f01a80b4d60 addr v1:172.21.15.204:6789/0 2024-03-25T18:24:22.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.617+0000 7f01afe4a640 10 monclient(hunting): picked mon.noname-b con 0x7f01a80b1730 addr v1:172.21.15.122:6790/0 2024-03-25T18:24:22.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.617+0000 7f01afe4a640 10 monclient(hunting): picked mon.noname-a con 0x7f01a814be90 addr v1:172.21.15.122:6789/0 2024-03-25T18:24:22.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.617+0000 7f01afe4a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80af940 con 0x7f01a814be90 2024-03-25T18:24:22.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.617+0000 7f01afe4a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80b0a70 con 0x7f01a80b4d60 2024-03-25T18:24:22.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.617+0000 7f01afe4a640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a80b0d50 con 0x7f01a80b1730 2024-03-25T18:24:22.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.617+0000 7f01afe4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:22.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.617+0000 7f01afe4a640 10 monclient(hunting): authenticate will time out at 7785.883620s 2024-03-25T18:24:22.628 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.627+0000 7f01adbe6640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f01a80b4d60 0x7f01a813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:50666/0 (socket says 172.21.15.122:50666) 2024-03-25T18:24:22.628 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.627+0000 7f01adbe6640 1 -- 172.21.15.122:0/2634136780 learned_addr learned my addr 172.21.15.122:0/2634136780 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:24:22.639 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.638+0000 7f01acbe4640 1 -- 172.21.15.122:0/2634136780 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 905896735 0 0) 0x7f01a80b0a70 con 0x7f01a80b4d60 2024-03-25T18:24:22.639 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.638+0000 7f01acbe4640 10 monclient(hunting): _negotiate 2024-03-25T18:24:22.639 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.638+0000 7f01acbe4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:22.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.638+0000 7f01acbe4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:22.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.638+0000 7f01acbe4640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:24:22.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.638+0000 7f01acbe4640 1 -- 172.21.15.122:0/2634136780 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0194003520 con 0x7f01a80b4d60 2024-03-25T18:24:22.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.639+0000 7f01acbe4640 1 -- 172.21.15.122:0/2634136780 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249611640 0 0) 0x7f01a80b0d50 con 0x7f01a80b1730 2024-03-25T18:24:22.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.639+0000 7f01acbe4640 10 monclient(hunting): _negotiate 2024-03-25T18:24:22.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.639+0000 7f01acbe4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:22.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.639+0000 7f01acbe4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:22.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.639+0000 7f01acbe4640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:24:22.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.639+0000 7f01acbe4640 1 -- 172.21.15.122:0/2634136780 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a80b0a70 con 0x7f01a80b1730 2024-03-25T18:24:22.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01acbe4640 1 -- 172.21.15.122:0/2634136780 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3368881998 0 0) 0x7f01a80b0a70 con 0x7f01a80b1730 2024-03-25T18:24:22.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01acbe4640 1 -- 172.21.15.122:0/2634136780 >> v1:172.21.15.204:6789/0 conn(0x7f01a80b4d60 legacy=0x7f01a813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:22.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01acbe4640 1 -- 172.21.15.122:0/2634136780 >> v1:172.21.15.122:6789/0 conn(0x7f01a814be90 legacy=0x7f01a814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:22.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01acbe4640 10 monclient: _finish_hunting 0 2024-03-25T18:24:22.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01acbe4640 1 monclient: found mon.noname-b 2024-03-25T18:24:22.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01acbe4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:22.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01acbe4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:24:22.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01acbe4640 1 -- 172.21.15.122:0/2634136780 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a80b1030 con 0x7f01a80b1730 2024-03-25T18:24:22.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01acbe4640 10 monclient: _finish_auth 0 2024-03-25T18:24:22.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01acbe4640 10 monclient: _check_auth_tickets 2024-03-25T18:24:22.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01acbe4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:22.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01afe4a640 5 monclient: authenticate success, global_id 4547 2024-03-25T18:24:22.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01afe4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:22.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01acbe4640 1 -- 172.21.15.122:0/2634136780 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f01a4001670 con 0x7f01a80b1730 2024-03-25T18:24:22.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01acbe4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:22.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01acbe4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:22.648 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01acbe4640 10 monclient: dump: 2024-03-25T18:24:22.648 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:22.648 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:22.648 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:22.648 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:22.648 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:22.648 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:22.648 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:22.648 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:22.649 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:22.649 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:22.649 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.645+0000 7f01afe4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:22.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.650+0000 7f01acbe4640 1 -- 172.21.15.122:0/2634136780 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01a4001890 con 0x7f01a80b1730 2024-03-25T18:24:22.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.650+0000 7f01acbe4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:22.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.650+0000 7f01acbe4640 1 -- 172.21.15.122:0/2634136780 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f01a4001ba0 con 0x7f01a80b1730 2024-03-25T18:24:22.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.650+0000 7f01afe4a640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:22.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.650+0000 7f01afe4a640 4 set_mon_vals no callback set 2024-03-25T18:24:22.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.650+0000 7f01acbe4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:22.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.650+0000 7f01acbe4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:22.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.650+0000 7f01acbe4640 10 monclient: dump: 2024-03-25T18:24:22.652 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:22.652 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:22.652 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:22.652 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:22.653 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:22.653 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:22.653 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:22.653 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:22.653 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:22.653 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:22.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.651+0000 7f01afe4a640 10 monclient: shutdown 2024-03-25T18:24:22.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.651+0000 7f01afe4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:22.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.651+0000 7f01afe4a640 1 -- 172.21.15.122:0/2634136780 >> v1:172.21.15.122:6790/0 conn(0x7f01a80b1730 legacy=0x7f01a80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:22.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.651+0000 7f01afe4a640 1 -- 172.21.15.122:0/2634136780 shutdown_connections 2024-03-25T18:24:22.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.651+0000 7f01afe4a640 1 -- 172.21.15.122:0/2634136780 >> 172.21.15.122:0/2634136780 conn(0x7f01a80617a0 msgr2=0x7f01a8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:22.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.651+0000 7f01afe4a640 1 -- 172.21.15.122:0/2634136780 shutdown_connections 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.651+0000 7f01afe4a640 1 -- 172.21.15.122:0/2634136780 wait complete. 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.652+0000 7f01afe4a640 10 monclient: build_initial_monmap 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.652+0000 7f01afe4a640 10 monclient: monmap: 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:22.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.652+0000 7f01afe4a640 1 librados: starting msgr at 2024-03-25T18:24:22.656 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.652+0000 7f01afe4a640 1 librados: starting objecter 2024-03-25T18:24:22.656 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.653+0000 7f01afe4a640 1 Processor -- start 2024-03-25T18:24:22.656 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.653+0000 7f01afe4a640 1 -- start start 2024-03-25T18:24:22.656 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.653+0000 7f01afe4a640 1 librados: setting wanted keys 2024-03-25T18:24:22.656 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.653+0000 7f01afe4a640 1 librados: calling monclient init 2024-03-25T18:24:22.656 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.653+0000 7f01afe4a640 10 monclient: init 2024-03-25T18:24:22.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.653+0000 7f01afe4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:22.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.653+0000 7f01afe4a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:22.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.653+0000 7f01afe4a640 10 monclient(hunting): picked mon.noname-b con 0x7f01a80b1730 addr v1:172.21.15.204:6789/0 2024-03-25T18:24:22.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.653+0000 7f01afe4a640 10 monclient(hunting): picked mon.noname-a con 0x7f01a80b4d60 addr v1:172.21.15.122:6789/0 2024-03-25T18:24:22.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.653+0000 7f01afe4a640 10 monclient(hunting): picked mon.noname-c con 0x7f01a814be90 addr v1:172.21.15.122:6790/0 2024-03-25T18:24:22.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.653+0000 7f01afe4a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a8136480 con 0x7f01a80b4d60 2024-03-25T18:24:22.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.653+0000 7f01afe4a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a81380b0 con 0x7f01a80b1730 2024-03-25T18:24:22.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.653+0000 7f01afe4a640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01a81d5230 con 0x7f01a814be90 2024-03-25T18:24:22.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.653+0000 7f01afe4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:22.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.653+0000 7f01afe4a640 10 monclient(hunting): authenticate will time out at 7785.919812s 2024-03-25T18:24:22.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.669+0000 7f01ad3e5640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f01a80b4d60 0x7f01a812fcf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:60388/0 (socket says 172.21.15.122:60388) 2024-03-25T18:24:22.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.669+0000 7f01ad3e5640 1 -- 172.21.15.122:0/195144834 learned_addr learned my addr 172.21.15.122:0/195144834 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:24:22.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.669+0000 7f019a7fc640 10 client.?.objecter ms_handle_connect 0x7f01a80b4d60 2024-03-25T18:24:22.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.669+0000 7f019a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:22.686 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.684+0000 7f019a7fc640 10 client.?.objecter ms_handle_connect 0x7f01a80b1730 2024-03-25T18:24:22.686 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.684+0000 7f019a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:22.688 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.687+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231147149 0 0) 0x7f01a8136480 con 0x7f01a80b4d60 2024-03-25T18:24:22.688 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.687+0000 7f019a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:22.688 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.687+0000 7f019a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:22.688 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.687+0000 7f019a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:22.688 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.687+0000 7f019a7fc640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:24:22.688 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.687+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f018c004eb0 con 0x7f01a80b4d60 2024-03-25T18:24:22.694 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.693+0000 7f019a7fc640 10 client.?.objecter ms_handle_connect 0x7f01a814be90 2024-03-25T18:24:22.694 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.693+0000 7f019a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:22.704 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.703+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3098637650 0 0) 0x7f01a81380b0 con 0x7f01a80b1730 2024-03-25T18:24:22.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.703+0000 7f019a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:22.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.703+0000 7f019a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:22.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.703+0000 7f019a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:22.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.703+0000 7f019a7fc640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:24:22.705 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.703+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a8136480 con 0x7f01a80b1730 2024-03-25T18:24:22.706 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.705+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1891855742 0 0) 0x7f01a81d5230 con 0x7f01a814be90 2024-03-25T18:24:22.706 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.705+0000 7f019a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:22.706 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.705+0000 7f019a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:22.706 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.705+0000 7f019a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:22.706 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.705+0000 7f019a7fc640 10 monclient(hunting): my global_id is 4550 2024-03-25T18:24:22.706 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.705+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01a81380b0 con 0x7f01a814be90 2024-03-25T18:24:22.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.724+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 186679291 0 0) 0x7f01a81380b0 con 0x7f01a814be90 2024-03-25T18:24:22.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.724+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01a81d5230 con 0x7f01a814be90 2024-03-25T18:24:22.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.724+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f01a0001a90 con 0x7f01a814be90 2024-03-25T18:24:22.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.724+0000 7f019a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:22.726 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.724+0000 7f019a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:22.726 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.724+0000 7f019a7fc640 10 monclient(hunting): dump: 2024-03-25T18:24:22.726 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:22.726 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:22.726 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:22.726 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:22.726 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:22.726 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:22.726 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:22.726 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:22.726 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:22.727 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:22.727 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.724+0000 7f019a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:22.731 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.730+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 896107289 0 0) 0x7f018c004eb0 con 0x7f01a80b4d60 2024-03-25T18:24:22.731 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.730+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01a81380b0 con 0x7f01a80b4d60 2024-03-25T18:24:22.732 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.730+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f01a4001c60 con 0x7f01a80b4d60 2024-03-25T18:24:22.732 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.730+0000 7f019a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:22.732 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.730+0000 7f019a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:22.732 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.730+0000 7f019a7fc640 10 monclient(hunting): dump: 2024-03-25T18:24:22.732 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:22.732 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:22.732 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:22.732 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:22.732 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:22.732 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:22.732 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:22.732 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:22.732 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:22.732 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:22.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1693590125 0 0) 0x7f01a81380b0 con 0x7f01a80b4d60 2024-03-25T18:24:22.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 >> v1:172.21.15.122:6790/0 conn(0x7f01a814be90 legacy=0x7f01a81d79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:22.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 >> v1:172.21.15.204:6789/0 conn(0x7f01a80b1730 legacy=0x7f01a8135d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:22.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f019a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:22.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f019a7fc640 1 monclient: found mon.a 2024-03-25T18:24:22.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f019a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:22.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f019a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:24:22.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01a8136e70 con 0x7f01a80b4d60 2024-03-25T18:24:22.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f019a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:22.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f019a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:22.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f019a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:22.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f01afe4a640 5 monclient: authenticate success, global_id 4590 2024-03-25T18:24:22.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f01afe4a640 10 monclient: _renew_subs 2024-03-25T18:24:22.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f01afe4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:24:22.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f01afe4a640 1 -- 172.21.15.122:0/195144834 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01a8130400 con 0x7f01a80b4d60 2024-03-25T18:24:22.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f01afe4a640 10 client.4590.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:22.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f01afe4a640 10 monclient: _renew_subs 2024-03-25T18:24:22.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f01afe4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:24:22.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f01afe4a640 1 -- 172.21.15.122:0/195144834 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01a81d80e0 con 0x7f01a80b4d60 2024-03-25T18:24:22.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.748+0000 7f01afe4a640 1 librados: init done 2024-03-25T18:24:22.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.749+0000 7f01afe4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:22.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.749+0000 7f01afe4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:22.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.749+0000 7f01afe4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:24:22.752 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:22.749+0000 7f01afe4a640 1 -- 172.21.15.122:0/195144834 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01a8137f20 con 0x7f01a80b4d60 2024-03-25T18:24:23.059 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.058+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01a40038a0 con 0x7f01a80b4d60 2024-03-25T18:24:23.059 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.058+0000 7f019a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:23.059 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.058+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f01a4003cf0 con 0x7f01a80b4d60 2024-03-25T18:24:23.059 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.058+0000 7f019a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:23.059 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.058+0000 7f019a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:23.059 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.058+0000 7f019a7fc640 10 monclient: dump: 2024-03-25T18:24:23.060 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:23.060 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:23.060 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:23.060 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:23.060 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:23.060 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:23.060 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:23.060 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:23.060 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:23.060 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:23.060 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.058+0000 7f019bfff640 4 set_mon_vals no callback set 2024-03-25T18:24:23.061 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.058+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f01a4003f10 con 0x7f01a80b4d60 2024-03-25T18:24:23.061 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.059+0000 7f01adbe6640 10 monclient: get_auth_request con 0x7f018c06dfb0 auth_method 0 2024-03-25T18:24:23.061 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.059+0000 7f019a7fc640 10 client.4590.objecter ms_handle_connect 0x7f018c06dfb0 2024-03-25T18:24:23.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.069+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f01a40549e0 con 0x7f01a80b4d60 2024-03-25T18:24:23.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.069+0000 7f019a7fc640 10 client.4590.objecter ms_dispatch 0x7f01a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:23.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.069+0000 7f019a7fc640 3 client.4590.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:23.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.069+0000 7f019a7fc640 3 client.4590.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:23.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.069+0000 7f019a7fc640 20 client.4590.objecter dump_active .. 0 homeless 2024-03-25T18:24:23.071 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.069+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f01a40504f0 con 0x7f01a80b4d60 2024-03-25T18:24:23.071 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.069+0000 7f019a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:23.071 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.069+0000 7f019a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:23.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.287+0000 7f01afe4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:23.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.287+0000 7f01afe4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:23.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.287+0000 7f01afe4a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:24:23.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.287+0000 7f01afe4a640 1 -- 172.21.15.122:0/195144834 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01a80627b0 con 0x7f01a80b4d60 2024-03-25T18:24:23.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.325+0000 7f019a7fc640 1 -- 172.21.15.122:0/195144834 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f01a4052420 con 0x7f01a80b4d60 2024-03-25T18:24:23.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.326+0000 7f019a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:23.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.326+0000 7f019a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:23.327 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:24:23.327 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:23.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.328+0000 7f01afe4a640 10 librados: watch_flush enter 2024-03-25T18:24:23.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.328+0000 7f01afe4a640 10 librados: watch_flush exit 2024-03-25T18:24:23.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.328+0000 7f01afe4a640 20 client.4590.objecter shutdown clearing up homeless session... 2024-03-25T18:24:23.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.328+0000 7f01afe4a640 10 client.4590.objecter successfully canceled tick 2024-03-25T18:24:23.330 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.328+0000 7f01afe4a640 1 -- 172.21.15.122:0/195144834 >> v1:172.21.15.204:6800/676228005 conn(0x7f018c06dfb0 legacy=0x7f018c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:23.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.328+0000 7f01afe4a640 10 monclient: shutdown 2024-03-25T18:24:23.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.328+0000 7f01afe4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:23.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.328+0000 7f01afe4a640 1 -- 172.21.15.122:0/195144834 >> v1:172.21.15.122:6789/0 conn(0x7f01a80b4d60 legacy=0x7f01a812fcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:23.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.328+0000 7f01afe4a640 1 -- 172.21.15.122:0/195144834 shutdown_connections 2024-03-25T18:24:23.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.328+0000 7f01afe4a640 1 -- 172.21.15.122:0/195144834 >> 172.21.15.122:0/195144834 conn(0x7f01a80617a0 msgr2=0x7f01a80b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:23.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.328+0000 7f01afe4a640 1 -- 172.21.15.122:0/195144834 shutdown_connections 2024-03-25T18:24:23.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.328+0000 7f01afe4a640 1 -- 172.21.15.122:0/195144834 wait complete. 2024-03-25T18:24:23.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:23.328+0000 7f01afe4a640 1 librados: shutdown 2024-03-25T18:24:23.342 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:30.149 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:30.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.259+0000 7f091c0e4640 10 monclient: get_monmap_and_config 2024-03-25T18:24:30.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.259+0000 7f091c0e4640 10 monclient: build_initial_monmap 2024-03-25T18:24:30.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.260+0000 7f091c0e4640 10 monclient: monmap: 2024-03-25T18:24:30.261 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:24:30.261 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:30.261 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:24:30.261104+0000 2024-03-25T18:24:30.261 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:24:30.261104+0000 2024-03-25T18:24:30.262 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:30.262 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:30.262 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:24:30.262 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:24:30.262 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:24:30.262 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:30.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.260+0000 7f091c0e4640 1 Processor -- start 2024-03-25T18:24:30.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.260+0000 7f091c0e4640 1 -- start start 2024-03-25T18:24:30.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.260+0000 7f091c0e4640 10 monclient: init 2024-03-25T18:24:30.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.261+0000 7f091c0e4640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:30.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.261+0000 7f091c0e4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:30.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.261+0000 7f091c0e4640 10 monclient(hunting): picked mon.noname-b con 0x7f09140a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:24:30.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.261+0000 7f091c0e4640 10 monclient(hunting): picked mon.noname-c con 0x7f091413f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:24:30.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.261+0000 7f091c0e4640 10 monclient(hunting): picked mon.noname-a con 0x7f09141430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:24:30.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.261+0000 7f091c0e4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0914142520 con 0x7f09141430c0 2024-03-25T18:24:30.263 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.261+0000 7f091c0e4640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09140b28a0 con 0x7f091413f710 2024-03-25T18:24:30.264 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.261+0000 7f091c0e4640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09140b2a30 con 0x7f09140a98f0 2024-03-25T18:24:30.264 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.261+0000 7f091c0e4640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:30.264 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.261+0000 7f091c0e4640 10 monclient(hunting): authenticate will time out at 7793.527455s 2024-03-25T18:24:30.269 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.268+0000 7f0919e80640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f09140a98f0 0x7f09140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:47282/0 (socket says 172.21.15.122:47282) 2024-03-25T18:24:30.269 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.268+0000 7f0919e80640 1 -- 172.21.15.122:0/2426631981 learned_addr learned my addr 172.21.15.122:0/2426631981 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:24:30.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.278+0000 7f0918e7e640 1 -- 172.21.15.122:0/2426631981 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580488209 0 0) 0x7f09140b2a30 con 0x7f09140a98f0 2024-03-25T18:24:30.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.278+0000 7f0918e7e640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.278+0000 7f0918e7e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.278+0000 7f0918e7e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.278+0000 7f0918e7e640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:24:30.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.278+0000 7f0918e7e640 1 -- 172.21.15.122:0/2426631981 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0900003520 con 0x7f09140a98f0 2024-03-25T18:24:30.285 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.284+0000 7f0918e7e640 1 -- 172.21.15.122:0/2426631981 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825239538 0 0) 0x7f09140b28a0 con 0x7f091413f710 2024-03-25T18:24:30.285 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.284+0000 7f0918e7e640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.285 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.284+0000 7f0918e7e640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.285 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.284+0000 7f0918e7e640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.285 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.284+0000 7f0918e7e640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:24:30.285 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.284+0000 7f0918e7e640 1 -- 172.21.15.122:0/2426631981 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09140b2a30 con 0x7f091413f710 2024-03-25T18:24:30.296 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.294+0000 7f0918e7e640 1 -- 172.21.15.122:0/2426631981 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2413340865 0 0) 0x7f0900003520 con 0x7f09140a98f0 2024-03-25T18:24:30.296 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.294+0000 7f0918e7e640 1 -- 172.21.15.122:0/2426631981 >> v1:172.21.15.204:6789/0 conn(0x7f091413f710 legacy=0x7f09141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.296 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.294+0000 7f0918e7e640 1 -- 172.21.15.122:0/2426631981 >> v1:172.21.15.122:6789/0 conn(0x7f09141430c0 legacy=0x7f091414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.296 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.294+0000 7f0918e7e640 10 monclient: _finish_hunting 0 2024-03-25T18:24:30.296 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.294+0000 7f0918e7e640 1 monclient: found mon.noname-b 2024-03-25T18:24:30.296 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.294+0000 7f0918e7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:30.296 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.294+0000 7f0918e7e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:24:30.296 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.294+0000 7f0918e7e640 1 -- 172.21.15.122:0/2426631981 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09140b44c0 con 0x7f09140a98f0 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.294+0000 7f0918e7e640 10 monclient: _finish_auth 0 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.294+0000 7f0918e7e640 10 monclient: _check_auth_tickets 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.294+0000 7f0918e7e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.294+0000 7f0918e7e640 1 -- 172.21.15.122:0/2426631981 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0910001690 con 0x7f09140a98f0 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.294+0000 7f0918e7e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.294+0000 7f0918e7e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.294+0000 7f0918e7e640 10 monclient: dump: 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:30.297 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:30.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.294+0000 7f091c0e4640 5 monclient: authenticate success, global_id 4553 2024-03-25T18:24:30.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.294+0000 7f091c0e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:30.309 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.308+0000 7f0918e7e640 1 -- 172.21.15.122:0/2426631981 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09100017f0 con 0x7f09140a98f0 2024-03-25T18:24:30.309 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.308+0000 7f0918e7e640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:30.309 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.308+0000 7f0918e7e640 1 -- 172.21.15.122:0/2426631981 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0910001c00 con 0x7f09140a98f0 2024-03-25T18:24:30.309 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.308+0000 7f0918e7e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.310 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.308+0000 7f0918e7e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:30.310 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.308+0000 7f0918e7e640 10 monclient: dump: 2024-03-25T18:24:30.310 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:30.310 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:30.310 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:30.310 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:30.310 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.310 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:30.310 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:30.310 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:30.310 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:30.310 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:30.310 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.308+0000 7f091c0e4640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:30.310 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.308+0000 7f091c0e4640 4 set_mon_vals no callback set 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.309+0000 7f091c0e4640 10 monclient: shutdown 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.309+0000 7f091c0e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.309+0000 7f091c0e4640 1 -- 172.21.15.122:0/2426631981 >> v1:172.21.15.122:6790/0 conn(0x7f09140a98f0 legacy=0x7f09140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.309+0000 7f091c0e4640 1 -- 172.21.15.122:0/2426631981 shutdown_connections 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.309+0000 7f091c0e4640 1 -- 172.21.15.122:0/2426631981 >> 172.21.15.122:0/2426631981 conn(0x7f09140b7ea0 msgr2=0x7f09140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.309+0000 7f091c0e4640 1 -- 172.21.15.122:0/2426631981 shutdown_connections 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.309+0000 7f091c0e4640 1 -- 172.21.15.122:0/2426631981 wait complete. 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.310+0000 7f091c0e4640 10 monclient: build_initial_monmap 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.310+0000 7f091c0e4640 10 monclient: monmap: 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:30.312 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.310+0000 7f091c0e4640 1 librados: starting msgr at 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.310+0000 7f091c0e4640 1 librados: starting objecter 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.310+0000 7f091c0e4640 1 Processor -- start 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.311+0000 7f091c0e4640 1 -- start start 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.311+0000 7f091c0e4640 1 librados: setting wanted keys 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.311+0000 7f091c0e4640 1 librados: calling monclient init 2024-03-25T18:24:30.313 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.311+0000 7f091c0e4640 10 monclient: init 2024-03-25T18:24:30.315 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.311+0000 7f091c0e4640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:30.315 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.311+0000 7f091c0e4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:30.315 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.311+0000 7f091c0e4640 10 monclient(hunting): picked mon.noname-c con 0x7f09140a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:24:30.315 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.312+0000 7f091c0e4640 10 monclient(hunting): picked mon.noname-a con 0x7f091413f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:24:30.315 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.312+0000 7f091c0e4640 10 monclient(hunting): picked mon.noname-b con 0x7f09141430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:24:30.315 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.312+0000 7f091c0e4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0914138b20 con 0x7f091413f710 2024-03-25T18:24:30.315 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.312+0000 7f091c0e4640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0914137a80 con 0x7f09141430c0 2024-03-25T18:24:30.315 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.312+0000 7f091c0e4640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0914139880 con 0x7f09140a98f0 2024-03-25T18:24:30.315 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.312+0000 7f091c0e4640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:30.315 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.312+0000 7f091c0e4640 10 monclient(hunting): authenticate will time out at 7793.578053s 2024-03-25T18:24:30.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.317+0000 7f091a681640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f09141430c0 0x7f09141e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:43436/0 (socket says 172.21.15.122:43436) 2024-03-25T18:24:30.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.317+0000 7f091a681640 1 -- 172.21.15.122:0/1209118106 learned_addr learned my addr 172.21.15.122:0/1209118106 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:24:30.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.320+0000 7f090a7fc640 10 client.?.objecter ms_handle_connect 0x7f09141430c0 2024-03-25T18:24:30.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.320+0000 7f090a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:30.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.328+0000 7f090a7fc640 10 client.?.objecter ms_handle_connect 0x7f09140a98f0 2024-03-25T18:24:30.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.328+0000 7f090a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:30.339 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.338+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729464334 0 0) 0x7f0914137a80 con 0x7f09141430c0 2024-03-25T18:24:30.339 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.338+0000 7f090a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.339 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.338+0000 7f090a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.339 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.338+0000 7f090a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.339 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.338+0000 7f090a7fc640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:24:30.339 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.338+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f8004eb0 con 0x7f09141430c0 2024-03-25T18:24:30.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.339+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92215099 0 0) 0x7f0914139880 con 0x7f09140a98f0 2024-03-25T18:24:30.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.339+0000 7f090a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.339+0000 7f090a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.339+0000 7f090a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.339+0000 7f090a7fc640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:24:30.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.339+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0914137a80 con 0x7f09140a98f0 2024-03-25T18:24:30.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.343+0000 7f090a7fc640 10 client.?.objecter ms_handle_connect 0x7f091413f710 2024-03-25T18:24:30.344 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.343+0000 7f090a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:30.352 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.351+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1604406347 0 0) 0x7f08f8004eb0 con 0x7f09141430c0 2024-03-25T18:24:30.352 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.351+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0914139880 con 0x7f09141430c0 2024-03-25T18:24:30.352 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.351+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f090c001c90 con 0x7f09141430c0 2024-03-25T18:24:30.352 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.351+0000 7f090a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.352 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.351+0000 7f090a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:30.352 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.351+0000 7f090a7fc640 10 monclient(hunting): dump: 2024-03-25T18:24:30.352 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:30.353 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:30.353 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:30.353 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:30.353 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.353 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:30.353 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:30.353 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:30.353 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:30.353 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:30.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.351+0000 7f090a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:30.353 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.352+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2321507517 0 0) 0x7f0914138b20 con 0x7f091413f710 2024-03-25T18:24:30.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.352+0000 7f090a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:30.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.352+0000 7f090a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:30.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.352+0000 7f090a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:30.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.352+0000 7f090a7fc640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:24:30.354 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.352+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08f8004eb0 con 0x7f091413f710 2024-03-25T18:24:30.359 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.357+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1562278217 0 0) 0x7f0914139880 con 0x7f09141430c0 2024-03-25T18:24:30.359 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.357+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 >> v1:172.21.15.122:6790/0 conn(0x7f09140a98f0 legacy=0x7f0914137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.359 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.357+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 >> v1:172.21.15.122:6789/0 conn(0x7f091413f710 legacy=0x7f0914139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:30.359 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.357+0000 7f090a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:30.359 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.357+0000 7f090a7fc640 1 monclient: found mon.b 2024-03-25T18:24:30.359 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.357+0000 7f090a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:30.359 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.357+0000 7f090a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:30.359 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.357+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0914139a10 con 0x7f09141430c0 2024-03-25T18:24:30.359 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.357+0000 7f090a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:30.359 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.357+0000 7f090a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:30.360 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.357+0000 7f090a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:30.360 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.357+0000 7f091c0e4640 5 monclient: authenticate success, global_id 4588 2024-03-25T18:24:30.360 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.357+0000 7f091c0e4640 10 monclient: _renew_subs 2024-03-25T18:24:30.360 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.357+0000 7f091c0e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:30.360 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.357+0000 7f091c0e4640 1 -- 172.21.15.122:0/1209118106 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09141dab00 con 0x7f09141430c0 2024-03-25T18:24:30.360 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.358+0000 7f091c0e4640 10 client.4588.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:30.360 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.358+0000 7f091c0e4640 10 monclient: _renew_subs 2024-03-25T18:24:30.360 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.358+0000 7f091c0e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:30.360 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.358+0000 7f091c0e4640 1 -- 172.21.15.122:0/1209118106 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f091413b4f0 con 0x7f09141430c0 2024-03-25T18:24:30.360 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.358+0000 7f091c0e4640 1 librados: init done 2024-03-25T18:24:30.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.358+0000 7f091c0e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:30.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.358+0000 7f091c0e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:30.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.358+0000 7f091c0e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:30.361 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.358+0000 7f091c0e4640 1 -- 172.21.15.122:0/1209118106 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09141dae00 con 0x7f09141430c0 2024-03-25T18:24:30.636 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.634+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f090c003990 con 0x7f09141430c0 2024-03-25T18:24:30.636 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.634+0000 7f090a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:30.636 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.634+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f090c003ce0 con 0x7f09141430c0 2024-03-25T18:24:30.636 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.634+0000 7f090a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:30.636 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.634+0000 7f090a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:30.636 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.634+0000 7f090a7fc640 10 monclient: dump: 2024-03-25T18:24:30.636 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:30.636 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:30.636 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:30.636 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:30.636 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:30.636 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:30.637 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:30.637 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:30.637 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:30.637 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:30.637 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.634+0000 7f090bfff640 4 set_mon_vals no callback set 2024-03-25T18:24:30.637 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.635+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f090c003f00 con 0x7f09141430c0 2024-03-25T18:24:30.637 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.636+0000 7f0919e80640 10 monclient: get_auth_request con 0x7f08f806de00 auth_method 0 2024-03-25T18:24:30.638 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.636+0000 7f090a7fc640 10 client.4588.objecter ms_handle_connect 0x7f08f806de00 2024-03-25T18:24:30.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.639+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f090c054b00 con 0x7f09141430c0 2024-03-25T18:24:30.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.639+0000 7f090a7fc640 10 client.4588.objecter ms_dispatch 0x7f0914000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:30.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.639+0000 7f090a7fc640 3 client.4588.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:30.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.639+0000 7f090a7fc640 3 client.4588.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:30.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.639+0000 7f090a7fc640 20 client.4588.objecter dump_active .. 0 homeless 2024-03-25T18:24:30.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.643+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f090c050610 con 0x7f09141430c0 2024-03-25T18:24:30.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.643+0000 7f090a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:30.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.643+0000 7f090a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:30.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.857+0000 7f091c0e4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:30.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.857+0000 7f091c0e4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:30.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.858+0000 7f091c0e4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:30.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:30.858+0000 7f091c0e4640 1 -- 172.21.15.122:0/1209118106 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09140b3e80 con 0x7f09141430c0 2024-03-25T18:24:31.028 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:31.027+0000 7f090a7fc640 1 -- 172.21.15.122:0/1209118106 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f090c052540 con 0x7f09141430c0 2024-03-25T18:24:31.028 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:31.027+0000 7f090a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:31.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:31.027+0000 7f090a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:31.029 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:24:31.029 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:31.031 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:31.030+0000 7f091c0e4640 10 librados: watch_flush enter 2024-03-25T18:24:31.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:31.030+0000 7f091c0e4640 10 librados: watch_flush exit 2024-03-25T18:24:31.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:31.030+0000 7f091c0e4640 20 client.4588.objecter shutdown clearing up homeless session... 2024-03-25T18:24:31.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:31.030+0000 7f091c0e4640 10 client.4588.objecter successfully canceled tick 2024-03-25T18:24:31.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:31.030+0000 7f091c0e4640 1 -- 172.21.15.122:0/1209118106 >> v1:172.21.15.204:6800/676228005 conn(0x7f08f806de00 legacy=0x7f08f80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:31.030+0000 7f091c0e4640 10 monclient: shutdown 2024-03-25T18:24:31.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:31.030+0000 7f091c0e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:31.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:31.030+0000 7f091c0e4640 1 -- 172.21.15.122:0/1209118106 >> v1:172.21.15.204:6789/0 conn(0x7f09141430c0 legacy=0x7f09141e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:31.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:31.030+0000 7f091c0e4640 1 -- 172.21.15.122:0/1209118106 shutdown_connections 2024-03-25T18:24:31.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:31.030+0000 7f091c0e4640 1 -- 172.21.15.122:0/1209118106 >> 172.21.15.122:0/1209118106 conn(0x7f09140b7ea0 msgr2=0x7f0914074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:31.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:31.030+0000 7f091c0e4640 1 -- 172.21.15.122:0/1209118106 shutdown_connections 2024-03-25T18:24:31.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:31.030+0000 7f091c0e4640 1 -- 172.21.15.122:0/1209118106 wait complete. 2024-03-25T18:24:31.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:31.030+0000 7f091c0e4640 1 librados: shutdown 2024-03-25T18:24:31.042 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:32.287 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:08:26.029 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:32.298 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:08:26.041 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:32.329 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:08:26.072 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:32.349 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:08:26.092 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:32.352 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:08:26.098 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:32.358 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:08:26.101 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:32.360 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:08:26.105 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:32.402 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:08:26.144 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:32.405 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:08:26.150 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:37.849 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:37.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.961+0000 7f57ce21e640 10 monclient: get_monmap_and_config 2024-03-25T18:24:37.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.961+0000 7f57ce21e640 10 monclient: build_initial_monmap 2024-03-25T18:24:37.962 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.961+0000 7f57ce21e640 10 monclient: monmap: 2024-03-25T18:24:37.962 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:24:37.962 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:37.962 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:24:37.962297+0000 2024-03-25T18:24:37.962 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:24:37.962297+0000 2024-03-25T18:24:37.963 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:37.963 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:37.963 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:24:37.963 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:24:37.963 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:24:37.963 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:37.963 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.962+0000 7f57ce21e640 1 Processor -- start 2024-03-25T18:24:37.963 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.962+0000 7f57ce21e640 1 -- start start 2024-03-25T18:24:37.963 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.962+0000 7f57ce21e640 10 monclient: init 2024-03-25T18:24:37.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.962+0000 7f57ce21e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:37.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.963+0000 7f57ce21e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:37.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.963+0000 7f57ce21e640 10 monclient(hunting): picked mon.noname-b con 0x7f57c80a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:24:37.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.963+0000 7f57ce21e640 10 monclient(hunting): picked mon.noname-c con 0x7f57c813f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:24:37.964 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.963+0000 7f57ce21e640 10 monclient(hunting): picked mon.noname-a con 0x7f57c81430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:24:37.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.963+0000 7f57ce21e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c8142520 con 0x7f57c81430c0 2024-03-25T18:24:37.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.963+0000 7f57ce21e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c80b28a0 con 0x7f57c813f710 2024-03-25T18:24:37.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.963+0000 7f57ce21e640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c80b2a30 con 0x7f57c80a98f0 2024-03-25T18:24:37.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.963+0000 7f57ce21e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:37.965 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.963+0000 7f57ce21e640 10 monclient(hunting): authenticate will time out at 7801.229086s 2024-03-25T18:24:37.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.969+0000 7f57bffff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f57c813f710 0x7f57c81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:48116/0 (socket says 172.21.15.122:48116) 2024-03-25T18:24:37.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.969+0000 7f57bffff640 1 -- 172.21.15.122:0/1080748827 learned_addr learned my addr 172.21.15.122:0/1080748827 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:24:37.983 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.981+0000 7f57c6ffd640 1 -- 172.21.15.122:0/1080748827 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576453746 0 0) 0x7f57c80b28a0 con 0x7f57c813f710 2024-03-25T18:24:37.983 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.981+0000 7f57c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:37.983 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.981+0000 7f57c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:37.983 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.981+0000 7f57c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:37.983 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.981+0000 7f57c6ffd640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:24:37.983 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.981+0000 7f57c6ffd640 1 -- 172.21.15.122:0/1080748827 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b4003520 con 0x7f57c813f710 2024-03-25T18:24:37.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.984+0000 7f57c6ffd640 1 -- 172.21.15.122:0/1080748827 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799438415 0 0) 0x7f57c80b2a30 con 0x7f57c80a98f0 2024-03-25T18:24:37.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.984+0000 7f57c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:37.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.984+0000 7f57c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:37.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.984+0000 7f57c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:37.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.984+0000 7f57c6ffd640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:24:37.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.985+0000 7f57c6ffd640 1 -- 172.21.15.122:0/1080748827 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b4003840 con 0x7f57c80a98f0 2024-03-25T18:24:37.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57c6ffd640 1 -- 172.21.15.122:0/1080748827 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4064775091 0 0) 0x7f57b4003840 con 0x7f57c80a98f0 2024-03-25T18:24:37.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57c6ffd640 1 -- 172.21.15.122:0/1080748827 >> v1:172.21.15.204:6789/0 conn(0x7f57c813f710 legacy=0x7f57c81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:37.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57c6ffd640 1 -- 172.21.15.122:0/1080748827 >> v1:172.21.15.122:6789/0 conn(0x7f57c81430c0 legacy=0x7f57c814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:37.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:37.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57c6ffd640 1 monclient: found mon.noname-b 2024-03-25T18:24:37.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:37.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57c6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:24:37.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57c6ffd640 1 -- 172.21.15.122:0/1080748827 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c80b44c0 con 0x7f57c80a98f0 2024-03-25T18:24:37.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57c6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:37.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:37.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:37.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57c6ffd640 1 -- 172.21.15.122:0/1080748827 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f57b8001670 con 0x7f57c80a98f0 2024-03-25T18:24:37.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57ce21e640 5 monclient: authenticate success, global_id 4559 2024-03-25T18:24:37.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57ce21e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:37.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:37.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57c6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:37.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57c6ffd640 10 monclient: dump: 2024-03-25T18:24:37.999 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:37.999 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:37.999 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:37.999 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:37.999 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:37.999 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:37.999 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:37.999 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:37.999 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:37.999 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:37.999 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:37.996+0000 7f57ce21e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:38.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.006+0000 7f57c6ffd640 1 -- 172.21.15.122:0/1080748827 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57b80017d0 con 0x7f57c80a98f0 2024-03-25T18:24:38.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.006+0000 7f57c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:38.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.006+0000 7f57c6ffd640 1 -- 172.21.15.122:0/1080748827 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f57b8001ba0 con 0x7f57c80a98f0 2024-03-25T18:24:38.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.006+0000 7f57c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:38.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.006+0000 7f57c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:38.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.006+0000 7f57c6ffd640 10 monclient: dump: 2024-03-25T18:24:38.008 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:38.008 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:38.008 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:38.008 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:38.008 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:38.008 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:38.008 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:38.008 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:38.008 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:38.008 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:38.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.006+0000 7f57ce21e640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:38.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.006+0000 7f57ce21e640 4 set_mon_vals no callback set 2024-03-25T18:24:38.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.007+0000 7f57ce21e640 10 monclient: shutdown 2024-03-25T18:24:38.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.007+0000 7f57ce21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:38.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.007+0000 7f57ce21e640 1 -- 172.21.15.122:0/1080748827 >> v1:172.21.15.122:6790/0 conn(0x7f57c80a98f0 legacy=0x7f57c80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.007+0000 7f57ce21e640 1 -- 172.21.15.122:0/1080748827 shutdown_connections 2024-03-25T18:24:38.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.007+0000 7f57ce21e640 1 -- 172.21.15.122:0/1080748827 >> 172.21.15.122:0/1080748827 conn(0x7f57c80b7ea0 msgr2=0x7f57c80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:38.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.007+0000 7f57ce21e640 1 -- 172.21.15.122:0/1080748827 shutdown_connections 2024-03-25T18:24:38.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.007+0000 7f57ce21e640 1 -- 172.21.15.122:0/1080748827 wait complete. 2024-03-25T18:24:38.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.008+0000 7f57ce21e640 10 monclient: build_initial_monmap 2024-03-25T18:24:38.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.008+0000 7f57ce21e640 10 monclient: monmap: 2024-03-25T18:24:38.011 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:24:38.011 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:38.011 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:24:38.011 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:24:38.011 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.008+0000 7f57ce21e640 1 librados: starting msgr at 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.008+0000 7f57ce21e640 1 librados: starting objecter 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.008+0000 7f57ce21e640 1 Processor -- start 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.008+0000 7f57ce21e640 1 -- start start 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.008+0000 7f57ce21e640 1 librados: setting wanted keys 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.008+0000 7f57ce21e640 1 librados: calling monclient init 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.008+0000 7f57ce21e640 10 monclient: init 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.009+0000 7f57ce21e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:38.012 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.009+0000 7f57ce21e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.009+0000 7f57ce21e640 10 monclient(hunting): picked mon.noname-c con 0x7f57c80a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.009+0000 7f57ce21e640 10 monclient(hunting): picked mon.noname-b con 0x7f57c813f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.009+0000 7f57ce21e640 10 monclient(hunting): picked mon.noname-a con 0x7f57c81430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.009+0000 7f57ce21e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c8136dd0 con 0x7f57c81430c0 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.009+0000 7f57ce21e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c8133a40 con 0x7f57c813f710 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.009+0000 7f57ce21e640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c80abde0 con 0x7f57c80a98f0 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.009+0000 7f57ce21e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:38.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.009+0000 7f57ce21e640 10 monclient(hunting): authenticate will time out at 7801.275379s 2024-03-25T18:24:38.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.012+0000 7f57bffff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f57c813f710 0x7f57c8132c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:48138/0 (socket says 172.21.15.122:48138) 2024-03-25T18:24:38.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.012+0000 7f57bffff640 1 -- 172.21.15.122:0/2418127978 learned_addr learned my addr 172.21.15.122:0/2418127978 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:24:38.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.013+0000 7f57c4ff9640 10 client.?.objecter ms_handle_connect 0x7f57c813f710 2024-03-25T18:24:38.014 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.013+0000 7f57c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:38.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.017+0000 7f57c4ff9640 10 client.?.objecter ms_handle_connect 0x7f57c80a98f0 2024-03-25T18:24:38.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.017+0000 7f57c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:38.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.018+0000 7f57c4ff9640 1 -- 172.21.15.122:0/2418127978 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2132487832 0 0) 0x7f57c8133a40 con 0x7f57c813f710 2024-03-25T18:24:38.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.018+0000 7f57c4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:38.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.018+0000 7f57c4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:38.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.018+0000 7f57c4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:38.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.018+0000 7f57c4ff9640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:24:38.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.018+0000 7f57c4ff9640 1 -- 172.21.15.122:0/2418127978 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57a8004eb0 con 0x7f57c813f710 2024-03-25T18:24:38.025 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.024+0000 7f57c4ff9640 10 client.?.objecter ms_handle_connect 0x7f57c81430c0 2024-03-25T18:24:38.025 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.024+0000 7f57c4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:38.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.025+0000 7f57c4ff9640 1 -- 172.21.15.122:0/2418127978 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2785573750 0 0) 0x7f57a8004eb0 con 0x7f57c813f710 2024-03-25T18:24:38.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.025+0000 7f57c4ff9640 1 -- 172.21.15.122:0/2418127978 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57c8133a40 con 0x7f57c813f710 2024-03-25T18:24:38.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.025+0000 7f57c4ff9640 1 -- 172.21.15.122:0/2418127978 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f57b0001990 con 0x7f57c813f710 2024-03-25T18:24:38.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.025+0000 7f57c4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:38.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.025+0000 7f57c4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:38.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.025+0000 7f57c4ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:38.027 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:38.028 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:38.028 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:38.028 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:38.028 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:38.028 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:38.028 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:38.028 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:38.028 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:38.028 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:38.028 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.025+0000 7f57c4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:38.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57c4ff9640 1 -- 172.21.15.122:0/2418127978 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 701924396 0 0) 0x7f57c8133a40 con 0x7f57c813f710 2024-03-25T18:24:38.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57c4ff9640 1 -- 172.21.15.122:0/2418127978 >> v1:172.21.15.122:6790/0 conn(0x7f57c80a98f0 legacy=0x7f57c81356b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57c4ff9640 1 -- 172.21.15.122:0/2418127978 >> v1:172.21.15.122:6789/0 conn(0x7f57c81430c0 legacy=0x7f57c8133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57c4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:24:38.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57c4ff9640 1 monclient: found mon.b 2024-03-25T18:24:38.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57c4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:38.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57c4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:38.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57c4ff9640 1 -- 172.21.15.122:0/2418127978 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c80ac0c0 con 0x7f57c813f710 2024-03-25T18:24:38.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57c4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:24:38.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57c4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:24:38.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57c4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:38.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57ce21e640 5 monclient: authenticate success, global_id 4594 2024-03-25T18:24:38.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57ce21e640 10 monclient: _renew_subs 2024-03-25T18:24:38.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57ce21e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:38.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57ce21e640 1 -- 172.21.15.122:0/2418127978 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c80ac3a0 con 0x7f57c813f710 2024-03-25T18:24:38.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57ce21e640 10 client.4594.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:38.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57ce21e640 10 monclient: _renew_subs 2024-03-25T18:24:38.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57ce21e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:38.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57ce21e640 1 -- 172.21.15.122:0/2418127978 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c8139260 con 0x7f57c813f710 2024-03-25T18:24:38.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.032+0000 7f57ce21e640 1 librados: init done 2024-03-25T18:24:38.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.033+0000 7f57ce21e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:38.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.033+0000 7f57ce21e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:38.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.033+0000 7f57ce21e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:38.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.033+0000 7f57ce21e640 1 -- 172.21.15.122:0/2418127978 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c8139560 con 0x7f57c813f710 2024-03-25T18:24:38.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.035+0000 7f57c4ff9640 1 -- 172.21.15.122:0/2418127978 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57b00047e0 con 0x7f57c813f710 2024-03-25T18:24:38.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.035+0000 7f57c4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:38.037 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.035+0000 7f57c67fc640 4 set_mon_vals no callback set 2024-03-25T18:24:38.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.314+0000 7f57c4ff9640 1 -- 172.21.15.122:0/2418127978 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f57b0004af0 con 0x7f57c813f710 2024-03-25T18:24:38.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.314+0000 7f57c4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:38.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.314+0000 7f57c4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:38.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.314+0000 7f57c4ff9640 10 monclient: dump: 2024-03-25T18:24:38.318 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:38.318 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:38.318 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:38.318 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:38.318 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:38.319 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:38.319 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:38.319 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:38.319 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:38.319 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:38.319 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.315+0000 7f57c4ff9640 1 -- 172.21.15.122:0/2418127978 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f57b001aac0 con 0x7f57c813f710 2024-03-25T18:24:38.319 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.315+0000 7f57c77fe640 10 monclient: get_auth_request con 0x7f57a806dd20 auth_method 0 2024-03-25T18:24:38.319 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.316+0000 7f57c4ff9640 10 client.4594.objecter ms_handle_connect 0x7f57a806dd20 2024-03-25T18:24:38.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.318+0000 7f57c4ff9640 1 -- 172.21.15.122:0/2418127978 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f57b0054900 con 0x7f57c813f710 2024-03-25T18:24:38.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.318+0000 7f57c4ff9640 10 client.4594.objecter ms_dispatch 0x7f57c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:38.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.318+0000 7f57c4ff9640 3 client.4594.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:38.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.318+0000 7f57c4ff9640 3 client.4594.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:38.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.318+0000 7f57c4ff9640 20 client.4594.objecter dump_active .. 0 homeless 2024-03-25T18:24:38.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.321+0000 7f57c4ff9640 1 -- 172.21.15.122:0/2418127978 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f57b0053fd0 con 0x7f57c813f710 2024-03-25T18:24:38.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.321+0000 7f57c4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:38.322 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.321+0000 7f57c4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:38.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.567+0000 7f57ce21e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:38.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.567+0000 7f57ce21e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:38.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.567+0000 7f57ce21e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:24:38.569 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.567+0000 7f57ce21e640 1 -- 172.21.15.122:0/2418127978 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57c80627b0 con 0x7f57c813f710 2024-03-25T18:24:38.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.585+0000 7f57c4ff9640 1 -- 172.21.15.122:0/2418127978 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f57b0050410 con 0x7f57c813f710 2024-03-25T18:24:38.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.585+0000 7f57c4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:38.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.585+0000 7f57c4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:38.587 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:24:38.587 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:38.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.587+0000 7f57ce21e640 10 librados: watch_flush enter 2024-03-25T18:24:38.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.587+0000 7f57ce21e640 10 librados: watch_flush exit 2024-03-25T18:24:38.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.587+0000 7f57ce21e640 20 client.4594.objecter shutdown clearing up homeless session... 2024-03-25T18:24:38.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.587+0000 7f57ce21e640 10 client.4594.objecter successfully canceled tick 2024-03-25T18:24:38.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.587+0000 7f57ce21e640 1 -- 172.21.15.122:0/2418127978 >> v1:172.21.15.204:6800/676228005 conn(0x7f57a806dd20 legacy=0x7f57a80701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.587+0000 7f57ce21e640 10 monclient: shutdown 2024-03-25T18:24:38.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.587+0000 7f57ce21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:38.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.587+0000 7f57ce21e640 1 -- 172.21.15.122:0/2418127978 >> v1:172.21.15.204:6789/0 conn(0x7f57c813f710 legacy=0x7f57c8132c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:38.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.587+0000 7f57ce21e640 1 -- 172.21.15.122:0/2418127978 shutdown_connections 2024-03-25T18:24:38.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.587+0000 7f57ce21e640 1 -- 172.21.15.122:0/2418127978 >> 172.21.15.122:0/2418127978 conn(0x7f57c80b7ea0 msgr2=0x7f57c813d4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:38.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.587+0000 7f57ce21e640 1 -- 172.21.15.122:0/2418127978 shutdown_connections 2024-03-25T18:24:38.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.587+0000 7f57ce21e640 1 -- 172.21.15.122:0/2418127978 wait complete. 2024-03-25T18:24:38.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:38.587+0000 7f57ce21e640 1 librados: shutdown 2024-03-25T18:24:38.601 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:45.407 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:45.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.523+0000 7f3cb1d40640 10 monclient: get_monmap_and_config 2024-03-25T18:24:45.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.523+0000 7f3cb1d40640 10 monclient: build_initial_monmap 2024-03-25T18:24:45.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.523+0000 7f3cb1d40640 10 monclient: monmap: 2024-03-25T18:24:45.523 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:24:45.524 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:45.524 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:24:45.523347+0000 2024-03-25T18:24:45.524 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:24:45.523347+0000 2024-03-25T18:24:45.524 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:45.524 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:45.524 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:24:45.524 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:24:45.524 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:24:45.524 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:45.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.524+0000 7f3cb1d40640 1 Processor -- start 2024-03-25T18:24:45.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.524+0000 7f3cb1d40640 1 -- start start 2024-03-25T18:24:45.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.524+0000 7f3cb1d40640 10 monclient: init 2024-03-25T18:24:45.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.524+0000 7f3cb1d40640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:45.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.524+0000 7f3cb1d40640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:45.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.524+0000 7f3cb1d40640 10 monclient(hunting): picked mon.noname-b con 0x7f3cac0a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:24:45.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.524+0000 7f3cb1d40640 10 monclient(hunting): picked mon.noname-c con 0x7f3cac13f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:24:45.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.524+0000 7f3cb1d40640 10 monclient(hunting): picked mon.noname-a con 0x7f3cac1430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:24:45.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.524+0000 7f3cb1d40640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac142520 con 0x7f3cac1430c0 2024-03-25T18:24:45.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.524+0000 7f3cb1d40640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac0b28a0 con 0x7f3cac13f710 2024-03-25T18:24:45.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.524+0000 7f3cb1d40640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac0b2a30 con 0x7f3cac0a98f0 2024-03-25T18:24:45.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.524+0000 7f3cb1d40640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:45.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.524+0000 7f3cb1d40640 10 monclient(hunting): authenticate will time out at 7808.789698s 2024-03-25T18:24:45.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.533+0000 7f3cab7fe640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f3cac0a98f0 0x7f3cac0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:46814/0 (socket says 172.21.15.122:46814) 2024-03-25T18:24:45.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.533+0000 7f3cab7fe640 1 -- 172.21.15.122:0/4192000600 learned_addr learned my addr 172.21.15.122:0/4192000600 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:24:45.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.546+0000 7f3caa7fc640 1 -- 172.21.15.122:0/4192000600 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3264370764 0 0) 0x7f3cac0b2a30 con 0x7f3cac0a98f0 2024-03-25T18:24:45.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.546+0000 7f3caa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.546+0000 7f3caa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.546+0000 7f3caa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.546+0000 7f3caa7fc640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:24:45.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.547+0000 7f3caa7fc640 1 -- 172.21.15.122:0/4192000600 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c94003520 con 0x7f3cac0a98f0 2024-03-25T18:24:45.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.553+0000 7f3caa7fc640 1 -- 172.21.15.122:0/4192000600 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 377400898 0 0) 0x7f3cac0b28a0 con 0x7f3cac13f710 2024-03-25T18:24:45.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.553+0000 7f3caa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.553+0000 7f3caa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.553+0000 7f3caa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.553+0000 7f3caa7fc640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:24:45.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.553+0000 7f3caa7fc640 1 -- 172.21.15.122:0/4192000600 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cac0b2a30 con 0x7f3cac13f710 2024-03-25T18:24:45.564 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.564+0000 7f3caa7fc640 1 -- 172.21.15.122:0/4192000600 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4143751403 0 0) 0x7f3cac0b2a30 con 0x7f3cac13f710 2024-03-25T18:24:45.564 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.564+0000 7f3caa7fc640 1 -- 172.21.15.122:0/4192000600 >> v1:172.21.15.122:6790/0 conn(0x7f3cac0a98f0 legacy=0x7f3cac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.564 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.564+0000 7f3caa7fc640 1 -- 172.21.15.122:0/4192000600 >> v1:172.21.15.122:6789/0 conn(0x7f3cac1430c0 legacy=0x7f3cac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.564+0000 7f3caa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:45.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.564+0000 7f3caa7fc640 1 monclient: found mon.noname-c 2024-03-25T18:24:45.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.564+0000 7f3caa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:45.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.564+0000 7f3caa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:24:45.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.564+0000 7f3caa7fc640 1 -- 172.21.15.122:0/4192000600 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac0b44c0 con 0x7f3cac13f710 2024-03-25T18:24:45.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.564+0000 7f3caa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:45.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.564+0000 7f3caa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:45.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.564+0000 7f3caa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:45.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.564+0000 7f3caa7fc640 1 -- 172.21.15.122:0/4192000600 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f3ca0001670 con 0x7f3cac13f710 2024-03-25T18:24:45.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.564+0000 7f3caa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.564+0000 7f3caa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:45.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.564+0000 7f3caa7fc640 10 monclient: dump: 2024-03-25T18:24:45.565 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:45.566 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:45.566 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:45.566 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:45.566 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.566 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:45.566 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:45.566 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:45.566 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:45.566 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:45.566 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.564+0000 7f3cb1d40640 5 monclient: authenticate success, global_id 4597 2024-03-25T18:24:45.566 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.564+0000 7f3cb1d40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:45.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.573+0000 7f3caa7fc640 1 -- 172.21.15.122:0/4192000600 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ca00017d0 con 0x7f3cac13f710 2024-03-25T18:24:45.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.573+0000 7f3caa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:45.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.573+0000 7f3caa7fc640 1 -- 172.21.15.122:0/4192000600 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f3ca0001b60 con 0x7f3cac13f710 2024-03-25T18:24:45.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.573+0000 7f3caa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.574+0000 7f3caa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:45.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.574+0000 7f3caa7fc640 10 monclient: dump: 2024-03-25T18:24:45.574 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:45.574 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:45.574 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:45.575 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:45.575 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.575 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:45.575 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:45.575 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:45.575 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:45.575 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:45.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.574+0000 7f3cb1d40640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:45.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.574+0000 7f3cb1d40640 4 set_mon_vals no callback set 2024-03-25T18:24:45.576 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.574+0000 7f3cb1d40640 10 monclient: shutdown 2024-03-25T18:24:45.577 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.574+0000 7f3cb1d40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:45.577 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.574+0000 7f3cb1d40640 1 -- 172.21.15.122:0/4192000600 >> v1:172.21.15.204:6789/0 conn(0x7f3cac13f710 legacy=0x7f3cac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.577 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.574+0000 7f3cb1d40640 1 -- 172.21.15.122:0/4192000600 shutdown_connections 2024-03-25T18:24:45.577 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.574+0000 7f3cb1d40640 1 -- 172.21.15.122:0/4192000600 >> 172.21.15.122:0/4192000600 conn(0x7f3cac0b7ea0 msgr2=0x7f3cac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:45.577 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.574+0000 7f3cb1d40640 1 -- 172.21.15.122:0/4192000600 shutdown_connections 2024-03-25T18:24:45.577 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.575+0000 7f3cb1d40640 1 -- 172.21.15.122:0/4192000600 wait complete. 2024-03-25T18:24:45.577 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.575+0000 7f3cb1d40640 10 monclient: build_initial_monmap 2024-03-25T18:24:45.577 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.575+0000 7f3cb1d40640 10 monclient: monmap: 2024-03-25T18:24:45.577 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:24:45.577 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:45.577 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:24:45.577 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:24:45.577 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:45.577 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:24:45.577 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:24:45.578 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:24:45.578 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:24:45.578 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:45.578 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.576+0000 7f3cb1d40640 1 librados: starting msgr at 2024-03-25T18:24:45.578 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.576+0000 7f3cb1d40640 1 librados: starting objecter 2024-03-25T18:24:45.578 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.576+0000 7f3cb1d40640 1 Processor -- start 2024-03-25T18:24:45.578 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.576+0000 7f3cb1d40640 1 -- start start 2024-03-25T18:24:45.578 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.576+0000 7f3cb1d40640 1 librados: setting wanted keys 2024-03-25T18:24:45.578 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.576+0000 7f3cb1d40640 1 librados: calling monclient init 2024-03-25T18:24:45.578 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.576+0000 7f3cb1d40640 10 monclient: init 2024-03-25T18:24:45.579 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.577+0000 7f3cb1d40640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:45.579 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.577+0000 7f3cb1d40640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:45.579 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.577+0000 7f3cb1d40640 10 monclient(hunting): picked mon.noname-b con 0x7f3cac0a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:24:45.579 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.577+0000 7f3cb1d40640 10 monclient(hunting): picked mon.noname-c con 0x7f3cac13f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:24:45.579 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.577+0000 7f3cb1d40640 10 monclient(hunting): picked mon.noname-a con 0x7f3cac1430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:24:45.579 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.577+0000 7f3cb1d40640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac1d7750 con 0x7f3cac1430c0 2024-03-25T18:24:45.579 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.577+0000 7f3cb1d40640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac136750 con 0x7f3cac0a98f0 2024-03-25T18:24:45.579 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.577+0000 7f3cb1d40640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cac1d85d0 con 0x7f3cac13f710 2024-03-25T18:24:45.579 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.577+0000 7f3cb1d40640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:45.579 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.577+0000 7f3cb1d40640 10 monclient(hunting): authenticate will time out at 7808.842430s 2024-03-25T18:24:45.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.594+0000 7f3caaffd640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f3cac13f710 0x7f3cac138300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:46842/0 (socket says 172.21.15.122:46842) 2024-03-25T18:24:45.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.594+0000 7f3caaffd640 1 -- 172.21.15.122:0/1488451127 learned_addr learned my addr 172.21.15.122:0/1488451127 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:24:45.601 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.601+0000 7f3c93fff640 10 client.?.objecter ms_handle_connect 0x7f3cac1430c0 2024-03-25T18:24:45.601 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.601+0000 7f3c93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:45.601 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.601+0000 7f3c93fff640 10 client.?.objecter ms_handle_connect 0x7f3cac13f710 2024-03-25T18:24:45.601 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.601+0000 7f3c93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:45.604 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.604+0000 7f3c93fff640 10 client.?.objecter ms_handle_connect 0x7f3cac0a98f0 2024-03-25T18:24:45.604 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.604+0000 7f3c93fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:45.607 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.607+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063338882 0 0) 0x7f3cac1d85d0 con 0x7f3cac13f710 2024-03-25T18:24:45.607 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.607+0000 7f3c93fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.607 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.607+0000 7f3c93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.607 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.607+0000 7f3c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.607 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.607+0000 7f3c93fff640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:24:45.607 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.607+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c98004eb0 con 0x7f3cac13f710 2024-03-25T18:24:45.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.614+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174579965 0 0) 0x7f3cac136750 con 0x7f3cac0a98f0 2024-03-25T18:24:45.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.614+0000 7f3c93fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.614+0000 7f3c93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.614+0000 7f3c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.614+0000 7f3c93fff640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:24:45.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.614+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cac1d85d0 con 0x7f3cac0a98f0 2024-03-25T18:24:45.616 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.616+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2059684825 0 0) 0x7f3cac1d7750 con 0x7f3cac1430c0 2024-03-25T18:24:45.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.616+0000 7f3c93fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:45.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.616+0000 7f3c93fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:45.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.616+0000 7f3c93fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:45.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.616+0000 7f3c93fff640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:24:45.617 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.616+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cac136750 con 0x7f3cac1430c0 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.627+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3838096368 0 0) 0x7f3c98004eb0 con 0x7f3cac13f710 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.627+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3cac1d7750 con 0x7f3cac13f710 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.627+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f3ca0001ba0 con 0x7f3cac13f710 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.627+0000 7f3c93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.627+0000 7f3c93fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.627+0000 7f3c93fff640 10 monclient(hunting): dump: 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:45.628 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.627+0000 7f3c93fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:45.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 60434040 0 0) 0x7f3cac136750 con 0x7f3cac1430c0 2024-03-25T18:24:45.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c98004eb0 con 0x7f3cac1430c0 2024-03-25T18:24:45.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f3ca4001ad0 con 0x7f3cac1430c0 2024-03-25T18:24:45.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3c93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3c93fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:45.640 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3c93fff640 10 monclient(hunting): dump: 2024-03-25T18:24:45.640 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:45.640 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:45.641 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:45.641 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:45.641 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.641 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:45.641 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:45.641 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:45.641 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:45.641 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:45.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2153013668 0 0) 0x7f3cac1d7750 con 0x7f3cac13f710 2024-03-25T18:24:45.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 >> v1:172.21.15.204:6789/0 conn(0x7f3cac0a98f0 legacy=0x7f3cac1d7ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 >> v1:172.21.15.122:6789/0 conn(0x7f3cac1430c0 legacy=0x7f3cac1d4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:45.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3c93fff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:45.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3c93fff640 1 monclient: found mon.c 2024-03-25T18:24:45.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3c93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:45.641 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3c93fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:24:45.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cac0ab950 con 0x7f3cac13f710 2024-03-25T18:24:45.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3c93fff640 10 monclient: _finish_auth 0 2024-03-25T18:24:45.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3c93fff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:45.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3c93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:45.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3cb1d40640 5 monclient: authenticate success, global_id 4568 2024-03-25T18:24:45.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3cb1d40640 10 monclient: _renew_subs 2024-03-25T18:24:45.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3cb1d40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:24:45.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3cb1d40640 1 -- 172.21.15.122:0/1488451127 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cac1d7a00 con 0x7f3cac13f710 2024-03-25T18:24:45.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3cb1d40640 10 client.4568.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:45.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3cb1d40640 10 monclient: _renew_subs 2024-03-25T18:24:45.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3cb1d40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:24:45.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3cb1d40640 1 -- 172.21.15.122:0/1488451127 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cac138e60 con 0x7f3cac13f710 2024-03-25T18:24:45.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.639+0000 7f3cb1d40640 1 librados: init done 2024-03-25T18:24:45.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.640+0000 7f3cb1d40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:45.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.640+0000 7f3cb1d40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:45.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.640+0000 7f3cb1d40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:24:45.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.640+0000 7f3cb1d40640 1 -- 172.21.15.122:0/1488451127 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cac1d7be0 con 0x7f3cac13f710 2024-03-25T18:24:45.952 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.951+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ca00038a0 con 0x7f3cac13f710 2024-03-25T18:24:45.952 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.951+0000 7f3c93fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:45.952 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.951+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f3ca0003b10 con 0x7f3cac13f710 2024-03-25T18:24:45.952 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.951+0000 7f3c93fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:45.952 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.951+0000 7f3c93fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:45.952 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.951+0000 7f3c93fff640 10 monclient: dump: 2024-03-25T18:24:45.953 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:45.953 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:45.953 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:45.953 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:45.953 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:45.953 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:45.953 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:45.953 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:45.953 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:45.953 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:45.953 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.951+0000 7f3ca9ffb640 4 set_mon_vals no callback set 2024-03-25T18:24:45.954 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.952+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f3ca0003d30 con 0x7f3cac13f710 2024-03-25T18:24:45.954 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.953+0000 7f3cab7fe640 10 monclient: get_auth_request con 0x7f3c9806df70 auth_method 0 2024-03-25T18:24:45.954 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.953+0000 7f3c93fff640 10 client.4568.objecter ms_handle_connect 0x7f3c9806df70 2024-03-25T18:24:45.961 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.960+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f3ca00547a0 con 0x7f3cac13f710 2024-03-25T18:24:45.961 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.960+0000 7f3c93fff640 10 client.4568.objecter ms_dispatch 0x7f3cac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:45.961 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.960+0000 7f3c93fff640 3 client.4568.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:45.961 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.960+0000 7f3c93fff640 3 client.4568.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:45.961 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.960+0000 7f3c93fff640 20 client.4568.objecter dump_active .. 0 homeless 2024-03-25T18:24:45.961 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.960+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3ca00502b0 con 0x7f3cac13f710 2024-03-25T18:24:45.961 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.960+0000 7f3c93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:45.961 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:45.960+0000 7f3c93fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:46.210 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.210+0000 7f3cb1d40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:46.210 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.210+0000 7f3cb1d40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:46.210 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.210+0000 7f3cb1d40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:24:46.211 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.210+0000 7f3cb1d40640 1 -- 172.21.15.122:0/1488451127 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3cac0627b0 con 0x7f3cac13f710 2024-03-25T18:24:46.250 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.249+0000 7f3c93fff640 1 -- 172.21.15.122:0/1488451127 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f3ca00521e0 con 0x7f3cac13f710 2024-03-25T18:24:46.250 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.249+0000 7f3c93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:46.250 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.249+0000 7f3c93fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:46.250 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:24:46.250 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.252+0000 7f3cb1d40640 10 librados: watch_flush enter 2024-03-25T18:24:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.252+0000 7f3cb1d40640 10 librados: watch_flush exit 2024-03-25T18:24:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.252+0000 7f3cb1d40640 20 client.4568.objecter shutdown clearing up homeless session... 2024-03-25T18:24:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.252+0000 7f3cb1d40640 10 client.4568.objecter successfully canceled tick 2024-03-25T18:24:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.252+0000 7f3cb1d40640 1 -- 172.21.15.122:0/1488451127 >> v1:172.21.15.204:6800/676228005 conn(0x7f3c9806df70 legacy=0x7f3c98070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.252+0000 7f3cb1d40640 10 monclient: shutdown 2024-03-25T18:24:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.252+0000 7f3cb1d40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.252+0000 7f3cb1d40640 1 -- 172.21.15.122:0/1488451127 >> v1:172.21.15.122:6790/0 conn(0x7f3cac13f710 legacy=0x7f3cac138300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.252+0000 7f3cb1d40640 1 -- 172.21.15.122:0/1488451127 shutdown_connections 2024-03-25T18:24:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.252+0000 7f3cb1d40640 1 -- 172.21.15.122:0/1488451127 >> 172.21.15.122:0/1488451127 conn(0x7f3cac0b7ea0 msgr2=0x7f3cac074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.252+0000 7f3cb1d40640 1 -- 172.21.15.122:0/1488451127 shutdown_connections 2024-03-25T18:24:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.252+0000 7f3cb1d40640 1 -- 172.21.15.122:0/1488451127 wait complete. 2024-03-25T18:24:46.253 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:46.252+0000 7f3cb1d40640 1 librados: shutdown 2024-03-25T18:24:46.264 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:52.216 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:08:45.958 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:52.225 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:08:45.967 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:52.269 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:08:46.012 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:52.282 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:08:46.027 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:52.287 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:08:46.029 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:52.288 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:08:46.034 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:52.296 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:08:46.039 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:52.320 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:08:46.065 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:52.325 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:08:46.068 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:53.070 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:53.185 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.184+0000 7f0954996640 10 monclient: get_monmap_and_config 2024-03-25T18:24:53.186 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.184+0000 7f0954996640 10 monclient: build_initial_monmap 2024-03-25T18:24:53.186 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.184+0000 7f0954996640 10 monclient: monmap: 2024-03-25T18:24:53.186 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:24:53.186 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:53.186 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:24:53.185800+0000 2024-03-25T18:24:53.186 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:24:53.185800+0000 2024-03-25T18:24:53.186 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:53.186 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:53.186 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:24:53.186 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:24:53.186 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:24:53.187 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:53.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.185+0000 7f0954996640 1 Processor -- start 2024-03-25T18:24:53.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.185+0000 7f0954996640 1 -- start start 2024-03-25T18:24:53.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.185+0000 7f0954996640 10 monclient: init 2024-03-25T18:24:53.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.186+0000 7f0954996640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:53.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.186+0000 7f0954996640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:53.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.186+0000 7f0954996640 10 monclient(hunting): picked mon.noname-b con 0x7f09500a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:24:53.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.186+0000 7f0954996640 10 monclient(hunting): picked mon.noname-a con 0x7f095013f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:24:53.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.186+0000 7f0954996640 10 monclient(hunting): picked mon.noname-c con 0x7f09501430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:24:53.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.186+0000 7f0954996640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0950142520 con 0x7f095013f710 2024-03-25T18:24:53.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.186+0000 7f0954996640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09500b28a0 con 0x7f09501430c0 2024-03-25T18:24:53.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.186+0000 7f0954996640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09500b2a30 con 0x7f09500a98f0 2024-03-25T18:24:53.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.186+0000 7f0954996640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:53.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.186+0000 7f0954996640 10 monclient(hunting): authenticate will time out at 7816.452517s 2024-03-25T18:24:53.195 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.194+0000 7f094e59c640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f09500a98f0 0x7f09500a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:51154/0 (socket says 172.21.15.122:51154) 2024-03-25T18:24:53.195 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.194+0000 7f094e59c640 1 -- 172.21.15.122:0/1400956414 learned_addr learned my addr 172.21.15.122:0/1400956414 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:24:53.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.206+0000 7f094dd9b640 1 -- 172.21.15.122:0/1400956414 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3569574501 0 0) 0x7f09500b28a0 con 0x7f09501430c0 2024-03-25T18:24:53.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.206+0000 7f094dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:53.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.206+0000 7f094dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:53.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.206+0000 7f094dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:53.207 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.206+0000 7f094dd9b640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:24:53.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.206+0000 7f094dd9b640 1 -- 172.21.15.122:0/1400956414 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f093c003520 con 0x7f09501430c0 2024-03-25T18:24:53.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.206+0000 7f094dd9b640 1 -- 172.21.15.122:0/1400956414 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519764946 0 0) 0x7f09500b2a30 con 0x7f09500a98f0 2024-03-25T18:24:53.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.206+0000 7f094dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:53.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.206+0000 7f094dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:53.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.206+0000 7f094dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:53.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.206+0000 7f094dd9b640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:24:53.208 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.207+0000 7f094dd9b640 1 -- 172.21.15.122:0/1400956414 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09500b28a0 con 0x7f09500a98f0 2024-03-25T18:24:53.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.214+0000 7f094dd9b640 1 -- 172.21.15.122:0/1400956414 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 434809472 0 0) 0x7f0950142520 con 0x7f095013f710 2024-03-25T18:24:53.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.214+0000 7f094dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:24:53.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.214+0000 7f094dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:53.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.214+0000 7f094dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:53.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.214+0000 7f094dd9b640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:24:53.216 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.214+0000 7f094dd9b640 1 -- 172.21.15.122:0/1400956414 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09500b2a30 con 0x7f095013f710 2024-03-25T18:24:53.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.235+0000 7f094dd9b640 1 -- 172.21.15.122:0/1400956414 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2017632827 0 0) 0x7f09500b2a30 con 0x7f095013f710 2024-03-25T18:24:53.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.235+0000 7f094dd9b640 1 -- 172.21.15.122:0/1400956414 >> v1:172.21.15.122:6790/0 conn(0x7f09500a98f0 legacy=0x7f09500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:53.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.235+0000 7f094dd9b640 1 -- 172.21.15.122:0/1400956414 >> v1:172.21.15.204:6789/0 conn(0x7f09501430c0 legacy=0x7f095014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:53.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.235+0000 7f094dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:24:53.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.235+0000 7f094dd9b640 1 monclient: found mon.noname-a 2024-03-25T18:24:53.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.235+0000 7f094dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:53.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.235+0000 7f094dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:24:53.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.235+0000 7f094dd9b640 1 -- 172.21.15.122:0/1400956414 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09500b44c0 con 0x7f095013f710 2024-03-25T18:24:53.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.235+0000 7f094dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:24:53.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.235+0000 7f094dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:24:53.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.235+0000 7f094dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:53.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.235+0000 7f094dd9b640 1 -- 172.21.15.122:0/1400956414 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f09380015b0 con 0x7f095013f710 2024-03-25T18:24:53.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.235+0000 7f094dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:53.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.235+0000 7f094dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:53.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.235+0000 7f094dd9b640 10 monclient: dump: 2024-03-25T18:24:53.238 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:53.238 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:53.238 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:53.238 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:53.238 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:53.238 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:53.238 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:53.238 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:53.238 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:53.238 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:53.239 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.235+0000 7f0954996640 5 monclient: authenticate success, global_id 4611 2024-03-25T18:24:53.239 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.235+0000 7f0954996640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:53.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.322+0000 7f094dd9b640 1 -- 172.21.15.122:0/1400956414 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0938001710 con 0x7f095013f710 2024-03-25T18:24:53.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.322+0000 7f094dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:53.323 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.322+0000 7f094dd9b640 1 -- 172.21.15.122:0/1400956414 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0938001b60 con 0x7f095013f710 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.322+0000 7f0954996640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.322+0000 7f0954996640 4 set_mon_vals no callback set 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.322+0000 7f094dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.322+0000 7f094dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.322+0000 7f094dd9b640 10 monclient: dump: 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:53.324 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:53.326 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.323+0000 7f0954996640 10 monclient: shutdown 2024-03-25T18:24:53.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.323+0000 7f0954996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:53.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.323+0000 7f0954996640 1 -- 172.21.15.122:0/1400956414 >> v1:172.21.15.122:6789/0 conn(0x7f095013f710 legacy=0x7f09501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:53.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.323+0000 7f0954996640 1 -- 172.21.15.122:0/1400956414 shutdown_connections 2024-03-25T18:24:53.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.323+0000 7f0954996640 1 -- 172.21.15.122:0/1400956414 >> 172.21.15.122:0/1400956414 conn(0x7f09500b7ea0 msgr2=0x7f09500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:53.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.323+0000 7f0954996640 1 -- 172.21.15.122:0/1400956414 shutdown_connections 2024-03-25T18:24:53.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.323+0000 7f0954996640 1 -- 172.21.15.122:0/1400956414 wait complete. 2024-03-25T18:24:53.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.324+0000 7f0954996640 10 monclient: build_initial_monmap 2024-03-25T18:24:53.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.324+0000 7f0954996640 10 monclient: monmap: 2024-03-25T18:24:53.327 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:24:53.327 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:53.327 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:24:53.327 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:24:53.327 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:53.328 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:24:53.328 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:24:53.328 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:24:53.328 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:24:53.328 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:53.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.324+0000 7f0954996640 1 librados: starting msgr at 2024-03-25T18:24:53.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.324+0000 7f0954996640 1 librados: starting objecter 2024-03-25T18:24:53.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.324+0000 7f0954996640 1 Processor -- start 2024-03-25T18:24:53.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.324+0000 7f0954996640 1 -- start start 2024-03-25T18:24:53.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.324+0000 7f0954996640 1 librados: setting wanted keys 2024-03-25T18:24:53.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.324+0000 7f0954996640 1 librados: calling monclient init 2024-03-25T18:24:53.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.324+0000 7f0954996640 10 monclient: init 2024-03-25T18:24:53.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.325+0000 7f0954996640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:53.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.325+0000 7f0954996640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:53.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.325+0000 7f0954996640 10 monclient(hunting): picked mon.noname-a con 0x7f09500a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:24:53.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.325+0000 7f0954996640 10 monclient(hunting): picked mon.noname-b con 0x7f095013f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:24:53.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.325+0000 7f0954996640 10 monclient(hunting): picked mon.noname-c con 0x7f09501430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:24:53.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.325+0000 7f0954996640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095013a570 con 0x7f09500a98f0 2024-03-25T18:24:53.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.325+0000 7f0954996640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f095013a0e0 con 0x7f095013f710 2024-03-25T18:24:53.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.325+0000 7f0954996640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09500b7c60 con 0x7f09501430c0 2024-03-25T18:24:53.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.325+0000 7f0954996640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:53.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.325+0000 7f0954996640 10 monclient(hunting): authenticate will time out at 7816.591455s 2024-03-25T18:24:53.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.329+0000 7f0947fff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f095013f710 0x7f095012d2d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:50154/0 (socket says 172.21.15.122:50154) 2024-03-25T18:24:53.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.329+0000 7f0947fff640 1 -- 172.21.15.122:0/4232657643 learned_addr learned my addr 172.21.15.122:0/4232657643 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:24:53.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.330+0000 7f0946ffd640 10 client.?.objecter ms_handle_connect 0x7f095013f710 2024-03-25T18:24:53.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.330+0000 7f0946ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:53.343 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.342+0000 7f0946ffd640 10 client.?.objecter ms_handle_connect 0x7f09500a98f0 2024-03-25T18:24:53.343 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.342+0000 7f0946ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:53.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.365+0000 7f0946ffd640 1 -- 172.21.15.122:0/4232657643 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188024006 0 0) 0x7f095013a570 con 0x7f09500a98f0 2024-03-25T18:24:53.366 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.365+0000 7f0946ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:53.367 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.365+0000 7f0946ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:53.367 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.365+0000 7f0946ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:53.367 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.365+0000 7f0946ffd640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:24:53.367 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.365+0000 7f0946ffd640 1 -- 172.21.15.122:0/4232657643 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0930004eb0 con 0x7f09500a98f0 2024-03-25T18:24:53.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.384+0000 7f0946ffd640 10 client.?.objecter ms_handle_connect 0x7f09501430c0 2024-03-25T18:24:53.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.384+0000 7f0946ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:53.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.385+0000 7f0946ffd640 1 -- 172.21.15.122:0/4232657643 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3761198728 0 0) 0x7f0930004eb0 con 0x7f09500a98f0 2024-03-25T18:24:53.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.385+0000 7f0946ffd640 1 -- 172.21.15.122:0/4232657643 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f095013a570 con 0x7f09500a98f0 2024-03-25T18:24:53.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.385+0000 7f0946ffd640 1 -- 172.21.15.122:0/4232657643 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0940001930 con 0x7f09500a98f0 2024-03-25T18:24:53.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.385+0000 7f0946ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:53.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.385+0000 7f0946ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:53.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.385+0000 7f0946ffd640 10 monclient(hunting): dump: 2024-03-25T18:24:53.387 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:53.387 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:53.387 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:53.387 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:53.387 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:53.387 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:53.387 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:53.387 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:53.387 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:53.387 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:53.387 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.385+0000 7f0946ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:53.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.416+0000 7f0946ffd640 1 -- 172.21.15.122:0/4232657643 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 16133048 0 0) 0x7f095013a570 con 0x7f09500a98f0 2024-03-25T18:24:53.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.416+0000 7f0946ffd640 1 -- 172.21.15.122:0/4232657643 >> v1:172.21.15.122:6790/0 conn(0x7f09501430c0 legacy=0x7f09501d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:53.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.416+0000 7f0946ffd640 1 -- 172.21.15.122:0/4232657643 >> v1:172.21.15.204:6789/0 conn(0x7f095013f710 legacy=0x7f095012d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:53.418 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.416+0000 7f0946ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:53.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.416+0000 7f0946ffd640 1 monclient: found mon.a 2024-03-25T18:24:53.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.416+0000 7f0946ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:53.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.416+0000 7f0946ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:24:53.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.416+0000 7f0946ffd640 1 -- 172.21.15.122:0/4232657643 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0950139d20 con 0x7f09500a98f0 2024-03-25T18:24:53.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.416+0000 7f0946ffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:53.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.416+0000 7f0946ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:53.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.416+0000 7f0946ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:53.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.417+0000 7f0954996640 5 monclient: authenticate success, global_id 4614 2024-03-25T18:24:53.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.417+0000 7f0954996640 10 monclient: _renew_subs 2024-03-25T18:24:53.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.417+0000 7f0954996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:24:53.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.417+0000 7f0954996640 1 -- 172.21.15.122:0/4232657643 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f095012d9e0 con 0x7f09500a98f0 2024-03-25T18:24:53.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.417+0000 7f0954996640 10 client.4614.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:53.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.417+0000 7f0954996640 10 monclient: _renew_subs 2024-03-25T18:24:53.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.417+0000 7f0954996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:24:53.419 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.417+0000 7f0954996640 1 -- 172.21.15.122:0/4232657643 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f095012f650 con 0x7f09500a98f0 2024-03-25T18:24:53.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.417+0000 7f0954996640 1 librados: init done 2024-03-25T18:24:53.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.417+0000 7f0954996640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:53.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.418+0000 7f0954996640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:53.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.418+0000 7f0954996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:24:53.420 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.418+0000 7f0954996640 1 -- 172.21.15.122:0/4232657643 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09500b3ef0 con 0x7f09500a98f0 2024-03-25T18:24:53.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.748+0000 7f0946ffd640 1 -- 172.21.15.122:0/4232657643 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09400047e0 con 0x7f09500a98f0 2024-03-25T18:24:53.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.748+0000 7f0946ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:53.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.748+0000 7f0946ffd640 1 -- 172.21.15.122:0/4232657643 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f0940004c30 con 0x7f09500a98f0 2024-03-25T18:24:53.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.748+0000 7f0946ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:53.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.748+0000 7f0946ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:53.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.748+0000 7f0946ffd640 10 monclient: dump: 2024-03-25T18:24:53.750 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:24:53.750 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:24:53.750 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:53.750 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:24:53.750 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:53.750 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:24:53.750 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:24:53.750 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:24:53.750 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:24:53.750 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:24:53.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.748+0000 7f094d59a640 4 set_mon_vals no callback set 2024-03-25T18:24:53.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.748+0000 7f0946ffd640 1 -- 172.21.15.122:0/4232657643 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f0940004e50 con 0x7f09500a98f0 2024-03-25T18:24:53.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.749+0000 7f0947fff640 10 monclient: get_auth_request con 0x7f093006dd00 auth_method 0 2024-03-25T18:24:53.751 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.749+0000 7f0946ffd640 10 client.4614.objecter ms_handle_connect 0x7f093006dd00 2024-03-25T18:24:53.754 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.753+0000 7f0946ffd640 1 -- 172.21.15.122:0/4232657643 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f0940054810 con 0x7f09500a98f0 2024-03-25T18:24:53.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.753+0000 7f0946ffd640 10 client.4614.objecter ms_dispatch 0x7f0950000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:53.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.753+0000 7f0946ffd640 3 client.4614.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:53.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.753+0000 7f0946ffd640 3 client.4614.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:53.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.753+0000 7f0946ffd640 20 client.4614.objecter dump_active .. 0 homeless 2024-03-25T18:24:53.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.753+0000 7f0946ffd640 1 -- 172.21.15.122:0/4232657643 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0940050320 con 0x7f09500a98f0 2024-03-25T18:24:53.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.753+0000 7f0946ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:53.755 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:53.753+0000 7f0946ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:54.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.006+0000 7f0954996640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:54.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.006+0000 7f0954996640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:54.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.007+0000 7f0954996640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:24:54.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.007+0000 7f0954996640 1 -- 172.21.15.122:0/4232657643 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09500627b0 con 0x7f09500a98f0 2024-03-25T18:24:54.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.047+0000 7f0946ffd640 1 -- 172.21.15.122:0/4232657643 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f0940052250 con 0x7f09500a98f0 2024-03-25T18:24:54.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.047+0000 7f0946ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:54.048 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.047+0000 7f0946ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:54.048 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:24:54.049 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:54.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.050+0000 7f0954996640 10 librados: watch_flush enter 2024-03-25T18:24:54.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.050+0000 7f0954996640 10 librados: watch_flush exit 2024-03-25T18:24:54.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.050+0000 7f0954996640 20 client.4614.objecter shutdown clearing up homeless session... 2024-03-25T18:24:54.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.050+0000 7f0954996640 10 client.4614.objecter successfully canceled tick 2024-03-25T18:24:54.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.050+0000 7f0954996640 1 -- 172.21.15.122:0/4232657643 >> v1:172.21.15.204:6800/676228005 conn(0x7f093006dd00 legacy=0x7f09300701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:54.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.050+0000 7f0954996640 10 monclient: shutdown 2024-03-25T18:24:54.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.050+0000 7f0954996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:54.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.050+0000 7f0954996640 1 -- 172.21.15.122:0/4232657643 >> v1:172.21.15.122:6789/0 conn(0x7f09500a98f0 legacy=0x7f09500b7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:54.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.050+0000 7f0954996640 1 -- 172.21.15.122:0/4232657643 shutdown_connections 2024-03-25T18:24:54.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.050+0000 7f0954996640 1 -- 172.21.15.122:0/4232657643 >> 172.21.15.122:0/4232657643 conn(0x7f09500b7ea0 msgr2=0x7f0950074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:54.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.050+0000 7f0954996640 1 -- 172.21.15.122:0/4232657643 shutdown_connections 2024-03-25T18:24:54.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.050+0000 7f0954996640 1 -- 172.21.15.122:0/4232657643 wait complete. 2024-03-25T18:24:54.052 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:24:54.050+0000 7f0954996640 1 librados: shutdown 2024-03-25T18:24:54.063 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:00.870 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:00.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.969+0000 7f9e6181e640 10 monclient: get_monmap_and_config 2024-03-25T18:25:00.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.969+0000 7f9e6181e640 10 monclient: build_initial_monmap 2024-03-25T18:25:00.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.969+0000 7f9e6181e640 10 monclient: monmap: 2024-03-25T18:25:00.971 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:25:00.971 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:00.971 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:25:00.970232+0000 2024-03-25T18:25:00.971 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:25:00.970232+0000 2024-03-25T18:25:00.971 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:00.971 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:00.971 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:25:00.971 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:25:00.971 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:25:00.971 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:00.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.969+0000 7f9e6181e640 1 Processor -- start 2024-03-25T18:25:00.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.969+0000 7f9e6181e640 1 -- start start 2024-03-25T18:25:00.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.969+0000 7f9e6181e640 10 monclient: init 2024-03-25T18:25:00.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.970+0000 7f9e6181e640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:00.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.970+0000 7f9e6181e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:00.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.970+0000 7f9e6181e640 10 monclient(hunting): picked mon.noname-c con 0x7f9e5c0a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:25:00.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.970+0000 7f9e6181e640 10 monclient(hunting): picked mon.noname-a con 0x7f9e5c13f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:25:00.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.970+0000 7f9e6181e640 10 monclient(hunting): picked mon.noname-b con 0x7f9e5c1430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:25:00.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.970+0000 7f9e6181e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c142520 con 0x7f9e5c13f710 2024-03-25T18:25:00.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.970+0000 7f9e6181e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c0b28a0 con 0x7f9e5c0a98f0 2024-03-25T18:25:00.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.970+0000 7f9e6181e640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c0b2a30 con 0x7f9e5c1430c0 2024-03-25T18:25:00.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.970+0000 7f9e6181e640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:00.973 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.970+0000 7f9e6181e640 10 monclient(hunting): authenticate will time out at 7824.236584s 2024-03-25T18:25:00.978 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.977+0000 7f9e5affd640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f9e5c0a98f0 0x7f9e5c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:54482/0 (socket says 172.21.15.122:54482) 2024-03-25T18:25:00.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.977+0000 7f9e5affd640 1 -- 172.21.15.122:0/3650473702 learned_addr learned my addr 172.21.15.122:0/3650473702 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:25:00.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.995+0000 7f9e59ffb640 1 -- 172.21.15.122:0/3650473702 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833233821 0 0) 0x7f9e5c0b28a0 con 0x7f9e5c0a98f0 2024-03-25T18:25:00.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.995+0000 7f9e59ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:00.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.995+0000 7f9e59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:00.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.995+0000 7f9e59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:00.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.995+0000 7f9e59ffb640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:25:00.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.995+0000 7f9e59ffb640 1 -- 172.21.15.122:0/3650473702 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e44003520 con 0x7f9e5c0a98f0 2024-03-25T18:25:00.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.996+0000 7f9e59ffb640 1 -- 172.21.15.122:0/3650473702 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2043136809 0 0) 0x7f9e5c0b2a30 con 0x7f9e5c1430c0 2024-03-25T18:25:00.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.996+0000 7f9e59ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:00.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.996+0000 7f9e59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:00.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.996+0000 7f9e59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:00.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.996+0000 7f9e59ffb640 10 monclient(hunting): my global_id is 4574 2024-03-25T18:25:00.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.996+0000 7f9e59ffb640 1 -- 172.21.15.122:0/3650473702 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e5c0b28a0 con 0x7f9e5c1430c0 2024-03-25T18:25:00.999 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.998+0000 7f9e59ffb640 1 -- 172.21.15.122:0/3650473702 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619801894 0 0) 0x7f9e5c142520 con 0x7f9e5c13f710 2024-03-25T18:25:00.999 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.998+0000 7f9e59ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:00.999 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.998+0000 7f9e59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:00.999 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.998+0000 7f9e59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:00.999 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.998+0000 7f9e59ffb640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:25:01.000 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:00.998+0000 7f9e59ffb640 1 -- 172.21.15.122:0/3650473702 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e5c0b2a30 con 0x7f9e5c13f710 2024-03-25T18:25:01.015 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.013+0000 7f9e59ffb640 1 -- 172.21.15.122:0/3650473702 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 47993712 0 0) 0x7f9e5c0b28a0 con 0x7f9e5c1430c0 2024-03-25T18:25:01.015 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.013+0000 7f9e59ffb640 1 -- 172.21.15.122:0/3650473702 >> v1:172.21.15.204:6789/0 conn(0x7f9e5c0a98f0 legacy=0x7f9e5c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:01.015 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.013+0000 7f9e59ffb640 1 -- 172.21.15.122:0/3650473702 >> v1:172.21.15.122:6789/0 conn(0x7f9e5c13f710 legacy=0x7f9e5c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:01.015 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.013+0000 7f9e59ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:01.016 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.013+0000 7f9e59ffb640 1 monclient: found mon.noname-b 2024-03-25T18:25:01.016 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.013+0000 7f9e59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:01.016 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.013+0000 7f9e59ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:25:01.016 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.013+0000 7f9e59ffb640 1 -- 172.21.15.122:0/3650473702 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c0b44c0 con 0x7f9e5c1430c0 2024-03-25T18:25:01.016 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.013+0000 7f9e59ffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:01.016 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.013+0000 7f9e59ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:01.016 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.013+0000 7f9e59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:01.016 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.013+0000 7f9e59ffb640 1 -- 172.21.15.122:0/3650473702 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9e540019f0 con 0x7f9e5c1430c0 2024-03-25T18:25:01.016 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.013+0000 7f9e59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:01.016 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.013+0000 7f9e59ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:01.016 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.013+0000 7f9e59ffb640 10 monclient: dump: 2024-03-25T18:25:01.016 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:01.016 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:01.016 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:01.017 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:01.017 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:01.017 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:01.017 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:01.017 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:01.017 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:01.017 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:01.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.013+0000 7f9e6181e640 5 monclient: authenticate success, global_id 4574 2024-03-25T18:25:01.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.013+0000 7f9e6181e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:01.050 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.049+0000 7f9e59ffb640 1 -- 172.21.15.122:0/3650473702 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e54001b50 con 0x7f9e5c1430c0 2024-03-25T18:25:01.050 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.049+0000 7f9e59ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:01.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.049+0000 7f9e59ffb640 1 -- 172.21.15.122:0/3650473702 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9e54001f60 con 0x7f9e5c1430c0 2024-03-25T18:25:01.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.049+0000 7f9e6181e640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:01.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.049+0000 7f9e6181e640 4 set_mon_vals no callback set 2024-03-25T18:25:01.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.049+0000 7f9e59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:01.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.049+0000 7f9e59ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:01.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.049+0000 7f9e59ffb640 10 monclient: dump: 2024-03-25T18:25:01.051 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:01.051 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:01.051 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:01.051 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:01.051 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:01.051 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:01.051 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:01.051 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:01.051 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:01.051 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:01.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.050+0000 7f9e6181e640 10 monclient: shutdown 2024-03-25T18:25:01.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.050+0000 7f9e6181e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:01.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.050+0000 7f9e6181e640 1 -- 172.21.15.122:0/3650473702 >> v1:172.21.15.122:6790/0 conn(0x7f9e5c1430c0 legacy=0x7f9e5c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:01.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.050+0000 7f9e6181e640 1 -- 172.21.15.122:0/3650473702 shutdown_connections 2024-03-25T18:25:01.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.050+0000 7f9e6181e640 1 -- 172.21.15.122:0/3650473702 >> 172.21.15.122:0/3650473702 conn(0x7f9e5c0b7ea0 msgr2=0x7f9e5c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:01.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.050+0000 7f9e6181e640 1 -- 172.21.15.122:0/3650473702 shutdown_connections 2024-03-25T18:25:01.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.050+0000 7f9e6181e640 1 -- 172.21.15.122:0/3650473702 wait complete. 2024-03-25T18:25:01.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.051+0000 7f9e6181e640 10 monclient: build_initial_monmap 2024-03-25T18:25:01.053 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.051+0000 7f9e6181e640 10 monclient: monmap: 2024-03-25T18:25:01.053 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:25:01.053 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:01.054 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:25:01.054 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:25:01.054 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:01.054 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:25:01.054 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:25:01.054 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:25:01.054 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:25:01.054 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:01.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.051+0000 7f9e6181e640 1 librados: starting msgr at 2024-03-25T18:25:01.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.051+0000 7f9e6181e640 1 librados: starting objecter 2024-03-25T18:25:01.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.051+0000 7f9e6181e640 1 Processor -- start 2024-03-25T18:25:01.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.051+0000 7f9e6181e640 1 -- start start 2024-03-25T18:25:01.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.051+0000 7f9e6181e640 1 librados: setting wanted keys 2024-03-25T18:25:01.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.051+0000 7f9e6181e640 1 librados: calling monclient init 2024-03-25T18:25:01.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.051+0000 7f9e6181e640 10 monclient: init 2024-03-25T18:25:01.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.052+0000 7f9e6181e640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:01.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.052+0000 7f9e6181e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:01.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.052+0000 7f9e6181e640 10 monclient(hunting): picked mon.noname-a con 0x7f9e5c0a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:25:01.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.052+0000 7f9e6181e640 10 monclient(hunting): picked mon.noname-c con 0x7f9e5c13f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:25:01.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.052+0000 7f9e6181e640 10 monclient(hunting): picked mon.noname-b con 0x7f9e5c1430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:25:01.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.052+0000 7f9e6181e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c0abac0 con 0x7f9e5c0a98f0 2024-03-25T18:25:01.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.052+0000 7f9e6181e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c1d4310 con 0x7f9e5c1430c0 2024-03-25T18:25:01.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.052+0000 7f9e6181e640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c138f10 con 0x7f9e5c13f710 2024-03-25T18:25:01.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.052+0000 7f9e6181e640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:01.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.052+0000 7f9e6181e640 10 monclient(hunting): authenticate will time out at 7824.318820s 2024-03-25T18:25:01.065 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.064+0000 7f9e5a7fc640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f9e5c13f710 0x7f9e5c134080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:39266/0 (socket says 172.21.15.122:39266) 2024-03-25T18:25:01.065 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.064+0000 7f9e5a7fc640 1 -- 172.21.15.122:0/1019986996 learned_addr learned my addr 172.21.15.122:0/1019986996 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:25:01.066 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.065+0000 7f9e43fff640 10 client.?.objecter ms_handle_connect 0x7f9e5c13f710 2024-03-25T18:25:01.066 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.065+0000 7f9e43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:01.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.072+0000 7f9e43fff640 10 client.?.objecter ms_handle_connect 0x7f9e5c1430c0 2024-03-25T18:25:01.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.072+0000 7f9e43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:01.076 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.075+0000 7f9e43fff640 10 client.?.objecter ms_handle_connect 0x7f9e5c0a98f0 2024-03-25T18:25:01.077 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.075+0000 7f9e43fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:01.083 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.082+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607433033 0 0) 0x7f9e5c138f10 con 0x7f9e5c13f710 2024-03-25T18:25:01.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.082+0000 7f9e43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:01.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.082+0000 7f9e43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:01.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.082+0000 7f9e43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:01.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.082+0000 7f9e43fff640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:25:01.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.082+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e48004eb0 con 0x7f9e5c13f710 2024-03-25T18:25:01.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.083+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120196687 0 0) 0x7f9e5c1d4310 con 0x7f9e5c1430c0 2024-03-25T18:25:01.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.083+0000 7f9e43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:01.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.083+0000 7f9e43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:01.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.083+0000 7f9e43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:01.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.083+0000 7f9e43fff640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:25:01.085 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.083+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e5c138f10 con 0x7f9e5c1430c0 2024-03-25T18:25:01.096 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.095+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3115478262 0 0) 0x7f9e5c138f10 con 0x7f9e5c1430c0 2024-03-25T18:25:01.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.095+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e5c1d4310 con 0x7f9e5c1430c0 2024-03-25T18:25:01.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.095+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9e54001e60 con 0x7f9e5c1430c0 2024-03-25T18:25:01.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.095+0000 7f9e43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:01.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.095+0000 7f9e43fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:01.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.095+0000 7f9e43fff640 10 monclient(hunting): dump: 2024-03-25T18:25:01.097 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:01.097 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:01.097 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:01.097 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:01.097 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:01.097 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:01.097 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:01.097 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:01.097 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:01.098 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:01.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.095+0000 7f9e43fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:01.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.101+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 545820702 0 0) 0x7f9e5c0abac0 con 0x7f9e5c0a98f0 2024-03-25T18:25:01.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.101+0000 7f9e43fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:01.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.101+0000 7f9e43fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:01.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.101+0000 7f9e43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:01.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.101+0000 7f9e43fff640 10 monclient(hunting): my global_id is 4620 2024-03-25T18:25:01.102 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.101+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e5c138f10 con 0x7f9e5c0a98f0 2024-03-25T18:25:01.108 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.106+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2218974953 0 0) 0x7f9e48004eb0 con 0x7f9e5c13f710 2024-03-25T18:25:01.108 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.106+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e5c0abac0 con 0x7f9e5c13f710 2024-03-25T18:25:01.108 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.106+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9e50001a10 con 0x7f9e5c13f710 2024-03-25T18:25:01.108 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.106+0000 7f9e43fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:01.108 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.106+0000 7f9e43fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:01.108 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.106+0000 7f9e43fff640 10 monclient(hunting): dump: 2024-03-25T18:25:01.108 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:01.108 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:01.108 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:01.108 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:01.108 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:01.108 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:01.108 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:01.109 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:01.109 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:01.109 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:01.114 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1084682812 0 0) 0x7f9e5c1d4310 con 0x7f9e5c1430c0 2024-03-25T18:25:01.114 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 >> v1:172.21.15.122:6790/0 conn(0x7f9e5c13f710 legacy=0x7f9e5c134080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:01.114 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 >> v1:172.21.15.122:6789/0 conn(0x7f9e5c0a98f0 legacy=0x7f9e5c138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:01.114 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e43fff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:01.114 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e43fff640 1 monclient: found mon.b 2024-03-25T18:25:01.114 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:01.114 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e43fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:25:01.114 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c135010 con 0x7f9e5c1430c0 2024-03-25T18:25:01.114 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e43fff640 10 monclient: _finish_auth 0 2024-03-25T18:25:01.114 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e43fff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:01.114 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:01.115 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e6181e640 5 monclient: authenticate success, global_id 4609 2024-03-25T18:25:01.115 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e6181e640 10 monclient: _renew_subs 2024-03-25T18:25:01.115 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e6181e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:25:01.115 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e6181e640 1 -- 172.21.15.122:0/1019986996 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5c134790 con 0x7f9e5c1430c0 2024-03-25T18:25:01.115 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e6181e640 10 client.4609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:01.115 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e6181e640 10 monclient: _renew_subs 2024-03-25T18:25:01.115 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e6181e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:25:01.115 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e6181e640 1 -- 172.21.15.122:0/1019986996 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5c1d5120 con 0x7f9e5c1430c0 2024-03-25T18:25:01.115 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.112+0000 7f9e6181e640 1 librados: init done 2024-03-25T18:25:01.116 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.113+0000 7f9e6181e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:01.116 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.113+0000 7f9e6181e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:01.116 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.113+0000 7f9e6181e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:25:01.116 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.113+0000 7f9e6181e640 1 -- 172.21.15.122:0/1019986996 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e5c1d44a0 con 0x7f9e5c1430c0 2024-03-25T18:25:01.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.393+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e54002040 con 0x7f9e5c1430c0 2024-03-25T18:25:01.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.393+0000 7f9e43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:01.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.393+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9e54003dc0 con 0x7f9e5c1430c0 2024-03-25T18:25:01.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.393+0000 7f9e43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:01.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.393+0000 7f9e43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:01.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.393+0000 7f9e43fff640 10 monclient: dump: 2024-03-25T18:25:01.395 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:01.395 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:01.395 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:01.395 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:01.396 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:01.396 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:01.396 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:01.396 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:01.396 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:01.396 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:01.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.393+0000 7f9e597fa640 4 set_mon_vals no callback set 2024-03-25T18:25:01.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.394+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f9e54019d90 con 0x7f9e5c1430c0 2024-03-25T18:25:01.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.395+0000 7f9e5affd640 10 monclient: get_auth_request con 0x7f9e4806df70 auth_method 0 2024-03-25T18:25:01.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.395+0000 7f9e43fff640 10 client.4609.objecter ms_handle_connect 0x7f9e4806df70 2024-03-25T18:25:01.402 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.401+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f9e54053900 con 0x7f9e5c1430c0 2024-03-25T18:25:01.402 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.401+0000 7f9e43fff640 10 client.4609.objecter ms_dispatch 0x7f9e5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:01.402 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.401+0000 7f9e43fff640 3 client.4609.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:01.402 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.401+0000 7f9e43fff640 3 client.4609.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:01.402 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.401+0000 7f9e43fff640 20 client.4609.objecter dump_active .. 0 homeless 2024-03-25T18:25:01.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.402+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9e5404f410 con 0x7f9e5c1430c0 2024-03-25T18:25:01.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.402+0000 7f9e43fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:01.403 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.402+0000 7f9e43fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:01.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.650+0000 7f9e6181e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:01.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.650+0000 7f9e6181e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:01.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.650+0000 7f9e6181e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:25:01.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.650+0000 7f9e6181e640 1 -- 172.21.15.122:0/1019986996 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e5c1351a0 con 0x7f9e5c1430c0 2024-03-25T18:25:01.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.672+0000 7f9e43fff640 1 -- 172.21.15.122:0/1019986996 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f9e54051340 con 0x7f9e5c1430c0 2024-03-25T18:25:01.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.672+0000 7f9e43fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:01.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.672+0000 7f9e43fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:01.673 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:25:01.674 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:01.676 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.674+0000 7f9e6181e640 10 librados: watch_flush enter 2024-03-25T18:25:01.676 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.674+0000 7f9e6181e640 10 librados: watch_flush exit 2024-03-25T18:25:01.676 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.674+0000 7f9e6181e640 20 client.4609.objecter shutdown clearing up homeless session... 2024-03-25T18:25:01.676 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.674+0000 7f9e6181e640 10 client.4609.objecter successfully canceled tick 2024-03-25T18:25:01.676 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.674+0000 7f9e6181e640 1 -- 172.21.15.122:0/1019986996 >> v1:172.21.15.204:6800/676228005 conn(0x7f9e4806df70 legacy=0x7f9e48070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:01.676 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.674+0000 7f9e6181e640 10 monclient: shutdown 2024-03-25T18:25:01.676 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.674+0000 7f9e6181e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:01.676 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.674+0000 7f9e6181e640 1 -- 172.21.15.122:0/1019986996 >> v1:172.21.15.204:6789/0 conn(0x7f9e5c1430c0 legacy=0x7f9e5c1d4830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:01.676 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.675+0000 7f9e6181e640 1 -- 172.21.15.122:0/1019986996 shutdown_connections 2024-03-25T18:25:01.676 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.675+0000 7f9e6181e640 1 -- 172.21.15.122:0/1019986996 >> 172.21.15.122:0/1019986996 conn(0x7f9e5c0b7ea0 msgr2=0x7f9e5c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:01.676 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.675+0000 7f9e6181e640 1 -- 172.21.15.122:0/1019986996 shutdown_connections 2024-03-25T18:25:01.677 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.675+0000 7f9e6181e640 1 -- 172.21.15.122:0/1019986996 wait complete. 2024-03-25T18:25:01.677 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:01.675+0000 7f9e6181e640 1 librados: shutdown 2024-03-25T18:25:01.689 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:08.496 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:08.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.609+0000 7f13e6b64640 10 monclient: get_monmap_and_config 2024-03-25T18:25:08.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.609+0000 7f13e6b64640 10 monclient: build_initial_monmap 2024-03-25T18:25:08.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.609+0000 7f13e6b64640 10 monclient: monmap: 2024-03-25T18:25:08.611 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:25:08.611 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:08.611 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:25:08.610965+0000 2024-03-25T18:25:08.611 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:25:08.610965+0000 2024-03-25T18:25:08.612 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:08.612 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:08.612 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:25:08.612 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:25:08.612 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:25:08.612 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:08.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.610+0000 7f13e6b64640 1 Processor -- start 2024-03-25T18:25:08.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.610+0000 7f13e6b64640 1 -- start start 2024-03-25T18:25:08.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.610+0000 7f13e6b64640 10 monclient: init 2024-03-25T18:25:08.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.611+0000 7f13e6b64640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:08.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.611+0000 7f13e6b64640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:08.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.611+0000 7f13e6b64640 10 monclient(hunting): picked mon.noname-a con 0x7f13e00a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:25:08.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.611+0000 7f13e6b64640 10 monclient(hunting): picked mon.noname-c con 0x7f13e013f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:25:08.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.611+0000 7f13e6b64640 10 monclient(hunting): picked mon.noname-b con 0x7f13e01430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:25:08.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.611+0000 7f13e6b64640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e0142520 con 0x7f13e00a98f0 2024-03-25T18:25:08.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.611+0000 7f13e6b64640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e00b28a0 con 0x7f13e013f710 2024-03-25T18:25:08.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.611+0000 7f13e6b64640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e00b2a30 con 0x7f13e01430c0 2024-03-25T18:25:08.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.611+0000 7f13e6b64640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:08.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.611+0000 7f13e6b64640 10 monclient(hunting): authenticate will time out at 7831.877942s 2024-03-25T18:25:08.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.620+0000 7f13e5101640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f13e01430c0 0x7f13e014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:43974/0 (socket says 172.21.15.122:43974) 2024-03-25T18:25:08.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.620+0000 7f13e5101640 1 -- 172.21.15.122:0/3239184984 learned_addr learned my addr 172.21.15.122:0/3239184984 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:25:08.631 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.630+0000 7f13df7fe640 1 -- 172.21.15.122:0/3239184984 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906304273 0 0) 0x7f13e00b2a30 con 0x7f13e01430c0 2024-03-25T18:25:08.631 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.630+0000 7f13df7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.631 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.630+0000 7f13df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.631 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.630+0000 7f13df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.631 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.630+0000 7f13df7fe640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:25:08.631 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.630+0000 7f13df7fe640 1 -- 172.21.15.122:0/3239184984 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d0003520 con 0x7f13e01430c0 2024-03-25T18:25:08.636 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.634+0000 7f13df7fe640 1 -- 172.21.15.122:0/3239184984 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590229934 0 0) 0x7f13e00b28a0 con 0x7f13e013f710 2024-03-25T18:25:08.636 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.634+0000 7f13df7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.636 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.634+0000 7f13df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.636 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.634+0000 7f13df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.636 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.634+0000 7f13df7fe640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:25:08.636 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.634+0000 7f13df7fe640 1 -- 172.21.15.122:0/3239184984 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13e00b2a30 con 0x7f13e013f710 2024-03-25T18:25:08.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13df7fe640 1 -- 172.21.15.122:0/3239184984 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3089746556 0 0) 0x7f13e00b2a30 con 0x7f13e013f710 2024-03-25T18:25:08.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13df7fe640 1 -- 172.21.15.122:0/3239184984 >> v1:172.21.15.122:6790/0 conn(0x7f13e01430c0 legacy=0x7f13e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13df7fe640 1 -- 172.21.15.122:0/3239184984 >> v1:172.21.15.122:6789/0 conn(0x7f13e00a98f0 legacy=0x7f13e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13df7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:08.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13df7fe640 1 monclient: found mon.noname-c 2024-03-25T18:25:08.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:08.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13df7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:25:08.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13df7fe640 1 -- 172.21.15.122:0/3239184984 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e00b44c0 con 0x7f13e013f710 2024-03-25T18:25:08.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13df7fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:08.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13df7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:08.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:08.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13df7fe640 1 -- 172.21.15.122:0/3239184984 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f13cc001670 con 0x7f13e013f710 2024-03-25T18:25:08.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13e6b64640 5 monclient: authenticate success, global_id 4612 2024-03-25T18:25:08.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13e6b64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:08.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13df7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:08.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13df7fe640 10 monclient: dump: 2024-03-25T18:25:08.646 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:08.646 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:08.646 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:08.646 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:08.646 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.647 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:08.647 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:08.647 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:08.647 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:08.647 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:08.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.644+0000 7f13e6b64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:08.649 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.648+0000 7f13df7fe640 1 -- 172.21.15.122:0/3239184984 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13cc0017d0 con 0x7f13e013f710 2024-03-25T18:25:08.649 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.648+0000 7f13df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:08.649 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.648+0000 7f13df7fe640 1 -- 172.21.15.122:0/3239184984 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f13cc001be0 con 0x7f13e013f710 2024-03-25T18:25:08.649 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.648+0000 7f13df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.649 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.648+0000 7f13df7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:08.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.648+0000 7f13df7fe640 10 monclient: dump: 2024-03-25T18:25:08.650 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:08.650 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:08.650 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:08.650 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:08.650 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.650 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:08.650 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:08.650 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:08.650 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:08.650 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:08.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.648+0000 7f13e6b64640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:08.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.648+0000 7f13e6b64640 4 set_mon_vals no callback set 2024-03-25T18:25:08.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.649+0000 7f13e6b64640 10 monclient: shutdown 2024-03-25T18:25:08.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.649+0000 7f13e6b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:08.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.649+0000 7f13e6b64640 1 -- 172.21.15.122:0/3239184984 >> v1:172.21.15.204:6789/0 conn(0x7f13e013f710 legacy=0x7f13e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.649+0000 7f13e6b64640 1 -- 172.21.15.122:0/3239184984 shutdown_connections 2024-03-25T18:25:08.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.649+0000 7f13e6b64640 1 -- 172.21.15.122:0/3239184984 >> 172.21.15.122:0/3239184984 conn(0x7f13e00b7ea0 msgr2=0x7f13e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:08.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.649+0000 7f13e6b64640 1 -- 172.21.15.122:0/3239184984 shutdown_connections 2024-03-25T18:25:08.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.649+0000 7f13e6b64640 1 -- 172.21.15.122:0/3239184984 wait complete. 2024-03-25T18:25:08.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.650+0000 7f13e6b64640 10 monclient: build_initial_monmap 2024-03-25T18:25:08.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.650+0000 7f13e6b64640 10 monclient: monmap: 2024-03-25T18:25:08.652 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:25:08.652 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:08.652 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:25:08.652 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:25:08.652 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:08.652 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:25:08.652 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:25:08.653 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:25:08.653 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:25:08.653 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:08.653 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.650+0000 7f13e6b64640 1 librados: starting msgr at 2024-03-25T18:25:08.653 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.650+0000 7f13e6b64640 1 librados: starting objecter 2024-03-25T18:25:08.653 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.650+0000 7f13e6b64640 1 Processor -- start 2024-03-25T18:25:08.653 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.650+0000 7f13e6b64640 1 -- start start 2024-03-25T18:25:08.653 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.650+0000 7f13e6b64640 1 librados: setting wanted keys 2024-03-25T18:25:08.653 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.650+0000 7f13e6b64640 1 librados: calling monclient init 2024-03-25T18:25:08.653 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.650+0000 7f13e6b64640 10 monclient: init 2024-03-25T18:25:08.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.651+0000 7f13e6b64640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:08.654 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.651+0000 7f13e6b64640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:08.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.651+0000 7f13e6b64640 10 monclient(hunting): picked mon.noname-a con 0x7f13e00a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:25:08.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.651+0000 7f13e6b64640 10 monclient(hunting): picked mon.noname-b con 0x7f13e013f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:25:08.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.651+0000 7f13e6b64640 10 monclient(hunting): picked mon.noname-c con 0x7f13e01430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:25:08.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.651+0000 7f13e6b64640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e00b1610 con 0x7f13e00a98f0 2024-03-25T18:25:08.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.652+0000 7f13e6b64640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e00b0b50 con 0x7f13e013f710 2024-03-25T18:25:08.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.652+0000 7f13e6b64640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e01d7b70 con 0x7f13e01430c0 2024-03-25T18:25:08.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.652+0000 7f13e6b64640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:08.655 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.652+0000 7f13e6b64640 10 monclient(hunting): authenticate will time out at 7831.918095s 2024-03-25T18:25:08.662 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.660+0000 7f13e5101640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f13e01430c0 0x7f13e01d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:44022/0 (socket says 172.21.15.122:44022) 2024-03-25T18:25:08.662 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.661+0000 7f13e5101640 1 -- 172.21.15.122:0/4189744081 learned_addr learned my addr 172.21.15.122:0/4189744081 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:25:08.662 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.661+0000 7f13dd7fa640 10 client.?.objecter ms_handle_connect 0x7f13e01430c0 2024-03-25T18:25:08.662 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.661+0000 7f13dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:08.663 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.662+0000 7f13dd7fa640 10 client.?.objecter ms_handle_connect 0x7f13e013f710 2024-03-25T18:25:08.663 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.662+0000 7f13dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:08.669 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.668+0000 7f13dd7fa640 10 client.?.objecter ms_handle_connect 0x7f13e00a98f0 2024-03-25T18:25:08.669 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.668+0000 7f13dd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:08.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.669+0000 7f13dd7fa640 1 -- 172.21.15.122:0/4189744081 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148371599 0 0) 0x7f13e01d7b70 con 0x7f13e01430c0 2024-03-25T18:25:08.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.669+0000 7f13dd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.669+0000 7f13dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.669+0000 7f13dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.669+0000 7f13dd7fa640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:25:08.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.669+0000 7f13dd7fa640 1 -- 172.21.15.122:0/4189744081 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c4004eb0 con 0x7f13e01430c0 2024-03-25T18:25:08.681 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.680+0000 7f13dd7fa640 1 -- 172.21.15.122:0/4189744081 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 263293447 0 0) 0x7f13c4004eb0 con 0x7f13e01430c0 2024-03-25T18:25:08.681 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.680+0000 7f13dd7fa640 1 -- 172.21.15.122:0/4189744081 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13e01d7b70 con 0x7f13e01430c0 2024-03-25T18:25:08.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.680+0000 7f13dd7fa640 1 -- 172.21.15.122:0/4189744081 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f13d8001bd0 con 0x7f13e01430c0 2024-03-25T18:25:08.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.680+0000 7f13dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:08.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.680+0000 7f13dd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:08.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.680+0000 7f13dd7fa640 10 monclient(hunting): dump: 2024-03-25T18:25:08.682 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:08.682 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:08.682 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:08.682 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:08.682 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:08.682 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:08.682 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:08.682 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:08.682 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:08.683 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:08.683 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.680+0000 7f13dd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:08.683 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.682+0000 7f13dd7fa640 1 -- 172.21.15.122:0/4189744081 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596879257 0 0) 0x7f13e00b0b50 con 0x7f13e013f710 2024-03-25T18:25:08.683 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.682+0000 7f13dd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:08.684 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.682+0000 7f13dd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:08.684 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.682+0000 7f13dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:08.684 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.682+0000 7f13dd7fa640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:25:08.684 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.682+0000 7f13dd7fa640 1 -- 172.21.15.122:0/4189744081 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13c4004eb0 con 0x7f13e013f710 2024-03-25T18:25:08.694 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.692+0000 7f13dd7fa640 1 -- 172.21.15.122:0/4189744081 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4111142072 0 0) 0x7f13e01d7b70 con 0x7f13e01430c0 2024-03-25T18:25:08.694 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.692+0000 7f13dd7fa640 1 -- 172.21.15.122:0/4189744081 >> v1:172.21.15.204:6789/0 conn(0x7f13e013f710 legacy=0x7f13e01d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.694 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.692+0000 7f13dd7fa640 1 -- 172.21.15.122:0/4189744081 >> v1:172.21.15.122:6789/0 conn(0x7f13e00a98f0 legacy=0x7f13e00b0f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:08.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.692+0000 7f13dd7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:25:08.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.692+0000 7f13dd7fa640 1 monclient: found mon.c 2024-03-25T18:25:08.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.692+0000 7f13dd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:08.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.692+0000 7f13dd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:08.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.692+0000 7f13dd7fa640 1 -- 172.21.15.122:0/4189744081 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e01d5000 con 0x7f13e01430c0 2024-03-25T18:25:08.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.692+0000 7f13dd7fa640 10 monclient: _finish_auth 0 2024-03-25T18:25:08.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.692+0000 7f13dd7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:25:08.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.692+0000 7f13dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:08.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.692+0000 7f13e6b64640 5 monclient: authenticate success, global_id 4583 2024-03-25T18:25:08.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.692+0000 7f13e6b64640 10 monclient: _renew_subs 2024-03-25T18:25:08.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.692+0000 7f13e6b64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:08.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.692+0000 7f13e6b64640 1 -- 172.21.15.122:0/4189744081 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e01d51f0 con 0x7f13e01430c0 2024-03-25T18:25:08.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.693+0000 7f13e6b64640 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:08.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.693+0000 7f13e6b64640 10 monclient: _renew_subs 2024-03-25T18:25:08.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.693+0000 7f13e6b64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:08.695 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.693+0000 7f13e6b64640 1 -- 172.21.15.122:0/4189744081 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e01d7840 con 0x7f13e01430c0 2024-03-25T18:25:08.696 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.693+0000 7f13e6b64640 1 librados: init done 2024-03-25T18:25:08.696 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.694+0000 7f13e6b64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:08.696 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.694+0000 7f13e6b64640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:08.696 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.694+0000 7f13e6b64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:08.696 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:08.694+0000 7f13e6b64640 1 -- 172.21.15.122:0/4189744081 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13e00627b0 con 0x7f13e01430c0 2024-03-25T18:25:09.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.001+0000 7f13dd7fa640 1 -- 172.21.15.122:0/4189744081 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13d8004b60 con 0x7f13e01430c0 2024-03-25T18:25:09.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.001+0000 7f13dd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:09.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.001+0000 7f13dd7fa640 1 -- 172.21.15.122:0/4189744081 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f13d8004f70 con 0x7f13e01430c0 2024-03-25T18:25:09.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.001+0000 7f13dd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:09.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.001+0000 7f13dd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:09.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.001+0000 7f13dd7fa640 10 monclient: dump: 2024-03-25T18:25:09.003 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:09.003 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:09.003 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:09.003 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:09.003 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:09.003 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:09.003 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:09.004 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:09.004 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:09.004 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:09.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.001+0000 7f13deffd640 4 set_mon_vals no callback set 2024-03-25T18:25:09.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.002+0000 7f13dd7fa640 1 -- 172.21.15.122:0/4189744081 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f13d8005190 con 0x7f13e01430c0 2024-03-25T18:25:09.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.002+0000 7f13e4900640 10 monclient: get_auth_request con 0x7f13c406dd40 auth_method 0 2024-03-25T18:25:09.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.003+0000 7f13dd7fa640 10 client.4583.objecter ms_handle_connect 0x7f13c406dd40 2024-03-25T18:25:09.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.008+0000 7f13dd7fa640 1 -- 172.21.15.122:0/4189744081 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f13d8054b70 con 0x7f13e01430c0 2024-03-25T18:25:09.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.008+0000 7f13dd7fa640 10 client.4583.objecter ms_dispatch 0x7f13e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:09.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.008+0000 7f13dd7fa640 3 client.4583.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:09.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.008+0000 7f13dd7fa640 3 client.4583.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:09.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.008+0000 7f13dd7fa640 20 client.4583.objecter dump_active .. 0 homeless 2024-03-25T18:25:09.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.008+0000 7f13dd7fa640 1 -- 172.21.15.122:0/4189744081 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f13d8050680 con 0x7f13e01430c0 2024-03-25T18:25:09.009 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.008+0000 7f13dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:09.010 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.008+0000 7f13dd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:09.229 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.228+0000 7f13e6b64640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:09.230 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.228+0000 7f13e6b64640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:09.230 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.228+0000 7f13e6b64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:09.230 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.228+0000 7f13e6b64640 1 -- 172.21.15.122:0/4189744081 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13e013c750 con 0x7f13e01430c0 2024-03-25T18:25:09.381 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.380+0000 7f13dd7fa640 1 -- 172.21.15.122:0/4189744081 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f13d80525b0 con 0x7f13e01430c0 2024-03-25T18:25:09.381 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.380+0000 7f13dd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:09.381 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.380+0000 7f13dd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:09.382 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:25:09.382 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:09.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.382+0000 7f13e6b64640 10 librados: watch_flush enter 2024-03-25T18:25:09.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.382+0000 7f13e6b64640 10 librados: watch_flush exit 2024-03-25T18:25:09.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.382+0000 7f13e6b64640 20 client.4583.objecter shutdown clearing up homeless session... 2024-03-25T18:25:09.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.382+0000 7f13e6b64640 10 client.4583.objecter successfully canceled tick 2024-03-25T18:25:09.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.382+0000 7f13e6b64640 1 -- 172.21.15.122:0/4189744081 >> v1:172.21.15.204:6800/676228005 conn(0x7f13c406dd40 legacy=0x7f13c4070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.383+0000 7f13e6b64640 10 monclient: shutdown 2024-03-25T18:25:09.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.383+0000 7f13e6b64640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:09.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.383+0000 7f13e6b64640 1 -- 172.21.15.122:0/4189744081 >> v1:172.21.15.122:6790/0 conn(0x7f13e01430c0 legacy=0x7f13e01d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:09.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.383+0000 7f13e6b64640 1 -- 172.21.15.122:0/4189744081 shutdown_connections 2024-03-25T18:25:09.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.383+0000 7f13e6b64640 1 -- 172.21.15.122:0/4189744081 >> 172.21.15.122:0/4189744081 conn(0x7f13e00b7ea0 msgr2=0x7f13e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:09.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.383+0000 7f13e6b64640 1 -- 172.21.15.122:0/4189744081 shutdown_connections 2024-03-25T18:25:09.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.383+0000 7f13e6b64640 1 -- 172.21.15.122:0/4189744081 wait complete. 2024-03-25T18:25:09.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:09.383+0000 7f13e6b64640 1 librados: shutdown 2024-03-25T18:25:09.396 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:12.274 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:09:06.016 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:12.287 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:09:06.029 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:12.321 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:09:06.063 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:12.351 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:09:06.096 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:12.352 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:09:06.094 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:12.362 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:09:06.107 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:12.362 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:09:06.104 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:12.384 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:09:06.130 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:12.393 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:09:06.136 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:16.203 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:16.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.316+0000 7faab95fd640 10 monclient: get_monmap_and_config 2024-03-25T18:25:16.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.316+0000 7faab95fd640 10 monclient: build_initial_monmap 2024-03-25T18:25:16.318 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.316+0000 7faab95fd640 10 monclient: monmap: 2024-03-25T18:25:16.318 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:25:16.318 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:16.318 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:25:16.317883+0000 2024-03-25T18:25:16.318 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:25:16.317883+0000 2024-03-25T18:25:16.318 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:16.319 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:16.319 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:25:16.319 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:25:16.319 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:25:16.319 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:16.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.317+0000 7faab95fd640 1 Processor -- start 2024-03-25T18:25:16.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.317+0000 7faab95fd640 1 -- start start 2024-03-25T18:25:16.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.317+0000 7faab95fd640 10 monclient: init 2024-03-25T18:25:16.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.318+0000 7faab95fd640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:16.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.318+0000 7faab95fd640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:16.320 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.318+0000 7faab95fd640 10 monclient(hunting): picked mon.noname-a con 0x7faab40a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:25:16.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.318+0000 7faab95fd640 10 monclient(hunting): picked mon.noname-b con 0x7faab413f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:25:16.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.318+0000 7faab95fd640 10 monclient(hunting): picked mon.noname-c con 0x7faab41430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:25:16.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.318+0000 7faab95fd640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab4142520 con 0x7faab40a98f0 2024-03-25T18:25:16.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.318+0000 7faab95fd640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab40b28a0 con 0x7faab41430c0 2024-03-25T18:25:16.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.318+0000 7faab95fd640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab40b2a30 con 0x7faab413f710 2024-03-25T18:25:16.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.318+0000 7faab95fd640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:16.321 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.318+0000 7faab95fd640 10 monclient(hunting): authenticate will time out at 7839.584521s 2024-03-25T18:25:16.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.332+0000 7faab37fe640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7faab41430c0 0x7faab414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:35314/0 (socket says 172.21.15.122:35314) 2024-03-25T18:25:16.334 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.332+0000 7faab37fe640 1 -- 172.21.15.122:0/3138647783 learned_addr learned my addr 172.21.15.122:0/3138647783 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:25:16.345 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.344+0000 7faab1ffb640 1 -- 172.21.15.122:0/3138647783 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860100131 0 0) 0x7faab40b2a30 con 0x7faab413f710 2024-03-25T18:25:16.345 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.344+0000 7faab1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.346 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.344+0000 7faab1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.346 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.344+0000 7faab1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.346 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.344+0000 7faab1ffb640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:25:16.346 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.344+0000 7faab1ffb640 1 -- 172.21.15.122:0/3138647783 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa9c003520 con 0x7faab413f710 2024-03-25T18:25:16.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.346+0000 7faab1ffb640 1 -- 172.21.15.122:0/3138647783 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683577426 0 0) 0x7faab40b28a0 con 0x7faab41430c0 2024-03-25T18:25:16.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.346+0000 7faab1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.346+0000 7faab1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.347 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.346+0000 7faab1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.346+0000 7faab1ffb640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:25:16.348 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.346+0000 7faab1ffb640 1 -- 172.21.15.122:0/3138647783 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab40b2a30 con 0x7faab41430c0 2024-03-25T18:25:16.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.362+0000 7faab1ffb640 1 -- 172.21.15.122:0/3138647783 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 743042231 0 0) 0x7faab4142520 con 0x7faab40a98f0 2024-03-25T18:25:16.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.362+0000 7faab1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.362+0000 7faab1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.362+0000 7faab1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.362+0000 7faab1ffb640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:25:16.363 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.362+0000 7faab1ffb640 1 -- 172.21.15.122:0/3138647783 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab40b28a0 con 0x7faab40a98f0 2024-03-25T18:25:16.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.368+0000 7faab1ffb640 1 -- 172.21.15.122:0/3138647783 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3740377972 0 0) 0x7faa9c003520 con 0x7faab413f710 2024-03-25T18:25:16.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.368+0000 7faab1ffb640 1 -- 172.21.15.122:0/3138647783 >> v1:172.21.15.204:6789/0 conn(0x7faab41430c0 legacy=0x7faab414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.368+0000 7faab1ffb640 1 -- 172.21.15.122:0/3138647783 >> v1:172.21.15.122:6789/0 conn(0x7faab40a98f0 legacy=0x7faab40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.368+0000 7faab1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:16.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.368+0000 7faab1ffb640 1 monclient: found mon.noname-b 2024-03-25T18:25:16.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.368+0000 7faab1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:16.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.368+0000 7faab1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:25:16.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.368+0000 7faab1ffb640 1 -- 172.21.15.122:0/3138647783 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab40b44c0 con 0x7faab413f710 2024-03-25T18:25:16.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.368+0000 7faab1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:16.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.368+0000 7faab1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:16.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.368+0000 7faab1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:16.371 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.368+0000 7faab1ffb640 1 -- 172.21.15.122:0/3138647783 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7faaa8001690 con 0x7faab413f710 2024-03-25T18:25:16.371 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.368+0000 7faab1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:16.371 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.368+0000 7faab1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:16.371 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.368+0000 7faab1ffb640 10 monclient: dump: 2024-03-25T18:25:16.371 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:16.371 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:16.371 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:16.371 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:16.371 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:16.371 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:16.371 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:16.371 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:16.371 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:16.371 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:16.371 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.368+0000 7faab95fd640 5 monclient: authenticate success, global_id 4586 2024-03-25T18:25:16.372 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.368+0000 7faab95fd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:16.390 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.389+0000 7faab1ffb640 1 -- 172.21.15.122:0/3138647783 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faaa80017f0 con 0x7faab413f710 2024-03-25T18:25:16.390 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.389+0000 7faab1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:16.390 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.389+0000 7faab1ffb640 1 -- 172.21.15.122:0/3138647783 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7faaa8001c00 con 0x7faab413f710 2024-03-25T18:25:16.391 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.389+0000 7faab95fd640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:16.391 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.389+0000 7faab95fd640 4 set_mon_vals no callback set 2024-03-25T18:25:16.391 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.389+0000 7faab1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:16.391 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.389+0000 7faab1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:16.391 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.389+0000 7faab1ffb640 10 monclient: dump: 2024-03-25T18:25:16.391 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:16.391 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:16.391 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:16.391 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:16.391 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:16.391 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:16.391 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:16.392 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:16.392 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:16.392 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:16.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.390+0000 7faab95fd640 10 monclient: shutdown 2024-03-25T18:25:16.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.390+0000 7faab95fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:16.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.390+0000 7faab95fd640 1 -- 172.21.15.122:0/3138647783 >> v1:172.21.15.122:6790/0 conn(0x7faab413f710 legacy=0x7faab41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.390+0000 7faab95fd640 1 -- 172.21.15.122:0/3138647783 shutdown_connections 2024-03-25T18:25:16.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.390+0000 7faab95fd640 1 -- 172.21.15.122:0/3138647783 >> 172.21.15.122:0/3138647783 conn(0x7faab40b7ea0 msgr2=0x7faab40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:16.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.390+0000 7faab95fd640 1 -- 172.21.15.122:0/3138647783 shutdown_connections 2024-03-25T18:25:16.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.390+0000 7faab95fd640 1 -- 172.21.15.122:0/3138647783 wait complete. 2024-03-25T18:25:16.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.391+0000 7faab95fd640 10 monclient: build_initial_monmap 2024-03-25T18:25:16.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.391+0000 7faab95fd640 10 monclient: monmap: 2024-03-25T18:25:16.395 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:25:16.395 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:16.395 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:25:16.395 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:25:16.395 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:16.395 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:25:16.395 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:25:16.395 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:25:16.395 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:25:16.395 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:16.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.391+0000 7faab95fd640 1 librados: starting msgr at 2024-03-25T18:25:16.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.391+0000 7faab95fd640 1 librados: starting objecter 2024-03-25T18:25:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.391+0000 7faab95fd640 1 Processor -- start 2024-03-25T18:25:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.391+0000 7faab95fd640 1 -- start start 2024-03-25T18:25:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.391+0000 7faab95fd640 1 librados: setting wanted keys 2024-03-25T18:25:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.391+0000 7faab95fd640 1 librados: calling monclient init 2024-03-25T18:25:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.391+0000 7faab95fd640 10 monclient: init 2024-03-25T18:25:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.392+0000 7faab95fd640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.392+0000 7faab95fd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.392+0000 7faab95fd640 10 monclient(hunting): picked mon.noname-b con 0x7faab40a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:25:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.392+0000 7faab95fd640 10 monclient(hunting): picked mon.noname-a con 0x7faab413f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:25:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.392+0000 7faab95fd640 10 monclient(hunting): picked mon.noname-c con 0x7faab41430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:25:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.392+0000 7faab95fd640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab40aba00 con 0x7faab413f710 2024-03-25T18:25:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.392+0000 7faab95fd640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab41d7490 con 0x7faab40a98f0 2024-03-25T18:25:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.392+0000 7faab95fd640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab41d8360 con 0x7faab41430c0 2024-03-25T18:25:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.392+0000 7faab95fd640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:16.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.392+0000 7faab95fd640 10 monclient(hunting): authenticate will time out at 7839.658635s 2024-03-25T18:25:16.402 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.401+0000 7faab27fc640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7faab413f710 0x7faab41d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:58770/0 (socket says 172.21.15.122:58770) 2024-03-25T18:25:16.402 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.401+0000 7faab27fc640 1 -- 172.21.15.122:0/3065899347 learned_addr learned my addr 172.21.15.122:0/3065899347 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:25:16.405 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.404+0000 7faa9b7fe640 10 client.?.objecter ms_handle_connect 0x7faab413f710 2024-03-25T18:25:16.405 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.404+0000 7faa9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:16.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.407+0000 7faa9b7fe640 10 client.?.objecter ms_handle_connect 0x7faab40a98f0 2024-03-25T18:25:16.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.407+0000 7faa9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:16.411 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.410+0000 7faa9b7fe640 10 client.?.objecter ms_handle_connect 0x7faab41430c0 2024-03-25T18:25:16.411 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.410+0000 7faa9b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:16.422 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.421+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977965491 0 0) 0x7faab41d7490 con 0x7faab40a98f0 2024-03-25T18:25:16.422 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.421+0000 7faa9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.422 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.421+0000 7faa9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.422 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.421+0000 7faa9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.422 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.421+0000 7faa9b7fe640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:25:16.423 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.421+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faaa0004eb0 con 0x7faab40a98f0 2024-03-25T18:25:16.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.431+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2802834161 0 0) 0x7faab41d8360 con 0x7faab41430c0 2024-03-25T18:25:16.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.431+0000 7faa9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.431+0000 7faa9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.431+0000 7faa9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.431+0000 7faa9b7fe640 10 monclient(hunting): my global_id is 4589 2024-03-25T18:25:16.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.431+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab41d7490 con 0x7faab41430c0 2024-03-25T18:25:16.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.432+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024940275 0 0) 0x7faab40aba00 con 0x7faab413f710 2024-03-25T18:25:16.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.432+0000 7faa9b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:16.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.432+0000 7faa9b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:16.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.432+0000 7faa9b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:16.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.432+0000 7faa9b7fe640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:25:16.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.432+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faab41d8360 con 0x7faab413f710 2024-03-25T18:25:16.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.443+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2865589673 0 0) 0x7faab41d7490 con 0x7faab41430c0 2024-03-25T18:25:16.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.443+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faab40aba00 con 0x7faab41430c0 2024-03-25T18:25:16.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.443+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7faaac001bd0 con 0x7faab41430c0 2024-03-25T18:25:16.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.443+0000 7faa9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:16.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.443+0000 7faa9b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:16.445 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.443+0000 7faa9b7fe640 10 monclient(hunting): dump: 2024-03-25T18:25:16.445 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:16.445 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:16.445 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:16.445 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:16.445 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:16.445 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:16.446 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:16.446 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:16.446 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:16.446 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:16.446 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.443+0000 7faa9b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:16.446 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.444+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3255812162 0 0) 0x7faaa0004eb0 con 0x7faab40a98f0 2024-03-25T18:25:16.446 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.444+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faab41d7490 con 0x7faab40a98f0 2024-03-25T18:25:16.446 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.444+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7faaa40018f0 con 0x7faab40a98f0 2024-03-25T18:25:16.447 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.444+0000 7faa9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:16.447 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.444+0000 7faa9b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:16.447 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.444+0000 7faa9b7fe640 10 monclient(hunting): dump: 2024-03-25T18:25:16.447 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:16.447 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:16.447 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:16.447 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:16.447 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:16.447 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:16.447 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:16.447 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:16.447 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:16.447 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:16.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.448+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1680624390 0 0) 0x7faab41d8360 con 0x7faab413f710 2024-03-25T18:25:16.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.448+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faaa0004eb0 con 0x7faab413f710 2024-03-25T18:25:16.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.448+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7faaa8001b00 con 0x7faab413f710 2024-03-25T18:25:16.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.448+0000 7faa9b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:16.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.448+0000 7faa9b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:16.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.448+0000 7faa9b7fe640 10 monclient(hunting): dump: 2024-03-25T18:25:16.450 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:16.450 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:16.450 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:16.450 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:16.450 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:16.450 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:16.450 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:16.450 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:16.450 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:16.450 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:16.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1076915776 0 0) 0x7faab41d7490 con 0x7faab40a98f0 2024-03-25T18:25:16.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 >> v1:172.21.15.122:6790/0 conn(0x7faab41430c0 legacy=0x7faab4133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 >> v1:172.21.15.122:6789/0 conn(0x7faab413f710 legacy=0x7faab41d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:16.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faa9b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:16.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faa9b7fe640 1 monclient: found mon.b 2024-03-25T18:25:16.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faa9b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:16.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faa9b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:25:16.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab40abc30 con 0x7faab40a98f0 2024-03-25T18:25:16.464 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faa9b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:16.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faa9b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:16.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faa9b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:16.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faab95fd640 5 monclient: authenticate success, global_id 4621 2024-03-25T18:25:16.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faab95fd640 10 monclient: _renew_subs 2024-03-25T18:25:16.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faab95fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:25:16.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faab95fd640 1 -- 172.21.15.122:0/3065899347 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab4137940 con 0x7faab40a98f0 2024-03-25T18:25:16.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faab95fd640 10 client.4621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:16.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faab95fd640 10 monclient: _renew_subs 2024-03-25T18:25:16.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faab95fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:25:16.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faab95fd640 1 -- 172.21.15.122:0/3065899347 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faab4135220 con 0x7faab40a98f0 2024-03-25T18:25:16.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.462+0000 7faab95fd640 1 librados: init done 2024-03-25T18:25:16.466 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.464+0000 7faab95fd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:16.466 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.464+0000 7faab95fd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:16.466 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.464+0000 7faab95fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:25:16.466 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.464+0000 7faab95fd640 1 -- 172.21.15.122:0/3065899347 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab40627b0 con 0x7faab40a98f0 2024-03-25T18:25:16.772 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.770+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faaa4003610 con 0x7faab40a98f0 2024-03-25T18:25:16.772 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.770+0000 7faa9b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:16.772 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.770+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7faaa4003920 con 0x7faab40a98f0 2024-03-25T18:25:16.773 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.771+0000 7faa9b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:16.773 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.771+0000 7faa9b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:16.773 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.771+0000 7faa9b7fe640 10 monclient: dump: 2024-03-25T18:25:16.773 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:16.773 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:16.773 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:16.773 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:16.773 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:16.773 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:16.773 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:16.773 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:16.773 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:16.773 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:16.773 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.771+0000 7faab17fa640 4 set_mon_vals no callback set 2024-03-25T18:25:16.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.771+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7faaa40198f0 con 0x7faab40a98f0 2024-03-25T18:25:16.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.772+0000 7faab27fc640 10 monclient: get_auth_request con 0x7faaa006e160 auth_method 0 2024-03-25T18:25:16.774 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.772+0000 7faa9b7fe640 10 client.4621.objecter ms_handle_connect 0x7faaa006e160 2024-03-25T18:25:16.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.778+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7faaa4053560 con 0x7faab40a98f0 2024-03-25T18:25:16.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.778+0000 7faa9b7fe640 10 client.4621.objecter ms_dispatch 0x7faab4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:16.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.778+0000 7faa9b7fe640 3 client.4621.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:16.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.778+0000 7faa9b7fe640 3 client.4621.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:16.779 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.778+0000 7faa9b7fe640 20 client.4621.objecter dump_active .. 0 homeless 2024-03-25T18:25:16.780 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.779+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faaa404f070 con 0x7faab40a98f0 2024-03-25T18:25:16.780 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.779+0000 7faa9b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:16.780 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.779+0000 7faa9b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:16.995 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.994+0000 7faab95fd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:16.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.994+0000 7faab95fd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:16.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.994+0000 7faab95fd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:25:16.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:16.994+0000 7faab95fd640 1 -- 172.21.15.122:0/3065899347 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faab41d7830 con 0x7faab40a98f0 2024-03-25T18:25:17.083 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:17.082+0000 7faa9b7fe640 1 -- 172.21.15.122:0/3065899347 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7faaa4050fa0 con 0x7faab40a98f0 2024-03-25T18:25:17.083 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:17.082+0000 7faa9b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:17.083 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:17.082+0000 7faa9b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:17.084 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:25:17.084 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:17.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:17.084+0000 7faab95fd640 10 librados: watch_flush enter 2024-03-25T18:25:17.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:17.084+0000 7faab95fd640 10 librados: watch_flush exit 2024-03-25T18:25:17.086 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:17.084+0000 7faab95fd640 20 client.4621.objecter shutdown clearing up homeless session... 2024-03-25T18:25:17.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:17.084+0000 7faab95fd640 10 client.4621.objecter successfully canceled tick 2024-03-25T18:25:17.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:17.084+0000 7faab95fd640 1 -- 172.21.15.122:0/3065899347 >> v1:172.21.15.204:6800/676228005 conn(0x7faaa006e160 legacy=0x7faaa0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:17.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:17.084+0000 7faab95fd640 10 monclient: shutdown 2024-03-25T18:25:17.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:17.084+0000 7faab95fd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:17.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:17.084+0000 7faab95fd640 1 -- 172.21.15.122:0/3065899347 >> v1:172.21.15.204:6789/0 conn(0x7faab40a98f0 legacy=0x7faab41386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:17.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:17.084+0000 7faab95fd640 1 -- 172.21.15.122:0/3065899347 shutdown_connections 2024-03-25T18:25:17.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:17.084+0000 7faab95fd640 1 -- 172.21.15.122:0/3065899347 >> 172.21.15.122:0/3065899347 conn(0x7faab40b7ea0 msgr2=0x7faab4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:17.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:17.084+0000 7faab95fd640 1 -- 172.21.15.122:0/3065899347 shutdown_connections 2024-03-25T18:25:17.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:17.084+0000 7faab95fd640 1 -- 172.21.15.122:0/3065899347 wait complete. 2024-03-25T18:25:17.087 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:17.084+0000 7faab95fd640 1 librados: shutdown 2024-03-25T18:25:17.098 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:23.905 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.016+0000 7ff8ffe25640 10 monclient: get_monmap_and_config 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.016+0000 7ff8ffe25640 10 monclient: build_initial_monmap 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.016+0000 7ff8ffe25640 10 monclient: monmap: 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:25:24.017748+0000 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:25:24.017748+0000 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:24.018 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:24.019 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:25:24.019 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:25:24.019 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:25:24.019 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:24.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.017+0000 7ff8ffe25640 1 Processor -- start 2024-03-25T18:25:24.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.017+0000 7ff8ffe25640 1 -- start start 2024-03-25T18:25:24.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.017+0000 7ff8ffe25640 10 monclient: init 2024-03-25T18:25:24.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.018+0000 7ff8ffe25640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:24.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.018+0000 7ff8ffe25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:24.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.018+0000 7ff8ffe25640 10 monclient(hunting): picked mon.noname-b con 0x7ff8f80a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:25:24.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.018+0000 7ff8ffe25640 10 monclient(hunting): picked mon.noname-a con 0x7ff8f813f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:25:24.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.018+0000 7ff8ffe25640 10 monclient(hunting): picked mon.noname-c con 0x7ff8f81430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:25:24.020 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.018+0000 7ff8ffe25640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f8142520 con 0x7ff8f813f710 2024-03-25T18:25:24.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.018+0000 7ff8ffe25640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f80b28a0 con 0x7ff8f81430c0 2024-03-25T18:25:24.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.018+0000 7ff8ffe25640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f80b2a30 con 0x7ff8f80a98f0 2024-03-25T18:25:24.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.018+0000 7ff8ffe25640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:24.021 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.018+0000 7ff8ffe25640 10 monclient(hunting): authenticate will time out at 7847.284529s 2024-03-25T18:25:24.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.033+0000 7ff8fe3c2640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7ff8f81430c0 0x7ff8f814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:48648/0 (socket says 172.21.15.122:48648) 2024-03-25T18:25:24.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.033+0000 7ff8fe3c2640 1 -- 172.21.15.122:0/424217455 learned_addr learned my addr 172.21.15.122:0/424217455 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:25:24.049 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.048+0000 7ff8fcbbf640 1 -- 172.21.15.122:0/424217455 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141638468 0 0) 0x7ff8f80b2a30 con 0x7ff8f80a98f0 2024-03-25T18:25:24.049 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.048+0000 7ff8fcbbf640 10 monclient(hunting): _negotiate 2024-03-25T18:25:24.049 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.048+0000 7ff8fcbbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:24.049 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.048+0000 7ff8fcbbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:24.050 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.048+0000 7ff8fcbbf640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:25:24.050 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.048+0000 7ff8fcbbf640 1 -- 172.21.15.122:0/424217455 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8e4003520 con 0x7ff8f80a98f0 2024-03-25T18:25:24.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.050+0000 7ff8fcbbf640 1 -- 172.21.15.122:0/424217455 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1623795870 0 0) 0x7ff8f80b28a0 con 0x7ff8f81430c0 2024-03-25T18:25:24.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.050+0000 7ff8fcbbf640 10 monclient(hunting): _negotiate 2024-03-25T18:25:24.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.050+0000 7ff8fcbbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:24.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.050+0000 7ff8fcbbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:24.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.050+0000 7ff8fcbbf640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:25:24.051 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.050+0000 7ff8fcbbf640 1 -- 172.21.15.122:0/424217455 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f80b2a30 con 0x7ff8f81430c0 2024-03-25T18:25:24.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.052+0000 7ff8fcbbf640 1 -- 172.21.15.122:0/424217455 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3773484986 0 0) 0x7ff8f8142520 con 0x7ff8f813f710 2024-03-25T18:25:24.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.052+0000 7ff8fcbbf640 10 monclient(hunting): _negotiate 2024-03-25T18:25:24.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.053+0000 7ff8fcbbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:24.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.053+0000 7ff8fcbbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:24.054 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.053+0000 7ff8fcbbf640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:25:24.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.053+0000 7ff8fcbbf640 1 -- 172.21.15.122:0/424217455 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f80b28a0 con 0x7ff8f813f710 2024-03-25T18:25:24.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.055+0000 7ff8fcbbf640 1 -- 172.21.15.122:0/424217455 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4031935677 0 0) 0x7ff8e4003520 con 0x7ff8f80a98f0 2024-03-25T18:25:24.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.055+0000 7ff8fcbbf640 1 -- 172.21.15.122:0/424217455 >> v1:172.21.15.204:6789/0 conn(0x7ff8f81430c0 legacy=0x7ff8f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.055+0000 7ff8fcbbf640 1 -- 172.21.15.122:0/424217455 >> v1:172.21.15.122:6789/0 conn(0x7ff8f813f710 legacy=0x7ff8f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.055+0000 7ff8fcbbf640 10 monclient: _finish_hunting 0 2024-03-25T18:25:24.056 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.055+0000 7ff8fcbbf640 1 monclient: found mon.noname-b 2024-03-25T18:25:24.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.055+0000 7ff8fcbbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:24.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.055+0000 7ff8fcbbf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:25:24.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.055+0000 7ff8fcbbf640 1 -- 172.21.15.122:0/424217455 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f80b44c0 con 0x7ff8f80a98f0 2024-03-25T18:25:24.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.055+0000 7ff8fcbbf640 10 monclient: _finish_auth 0 2024-03-25T18:25:24.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.055+0000 7ff8fcbbf640 10 monclient: _check_auth_tickets 2024-03-25T18:25:24.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.055+0000 7ff8fcbbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:24.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.055+0000 7ff8fcbbf640 1 -- 172.21.15.122:0/424217455 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff8f4001690 con 0x7ff8f80a98f0 2024-03-25T18:25:24.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.055+0000 7ff8fcbbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:24.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.055+0000 7ff8fcbbf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:24.057 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.055+0000 7ff8fcbbf640 10 monclient: dump: 2024-03-25T18:25:24.057 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:24.057 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:24.057 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:24.057 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:24.057 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:24.057 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:24.058 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:24.058 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:24.058 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:24.058 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:24.058 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.055+0000 7ff8ffe25640 5 monclient: authenticate success, global_id 4592 2024-03-25T18:25:24.058 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.055+0000 7ff8ffe25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:24.063 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.062+0000 7ff8fcbbf640 1 -- 172.21.15.122:0/424217455 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8f40017f0 con 0x7ff8f80a98f0 2024-03-25T18:25:24.063 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.062+0000 7ff8fcbbf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:24.064 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.062+0000 7ff8fcbbf640 1 -- 172.21.15.122:0/424217455 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff8f4001c00 con 0x7ff8f80a98f0 2024-03-25T18:25:24.064 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.062+0000 7ff8ffe25640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:24.064 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.062+0000 7ff8ffe25640 4 set_mon_vals no callback set 2024-03-25T18:25:24.064 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.062+0000 7ff8fcbbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:24.064 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.062+0000 7ff8fcbbf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:24.064 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.062+0000 7ff8fcbbf640 10 monclient: dump: 2024-03-25T18:25:24.064 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:24.064 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:24.064 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:24.064 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:24.065 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:24.065 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:24.065 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:24.065 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:24.065 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:24.065 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:24.068 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.062+0000 7ff8ffe25640 10 monclient: shutdown 2024-03-25T18:25:24.068 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.062+0000 7ff8ffe25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:24.068 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.062+0000 7ff8ffe25640 1 -- 172.21.15.122:0/424217455 >> v1:172.21.15.122:6790/0 conn(0x7ff8f80a98f0 legacy=0x7ff8f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.068 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.063+0000 7ff8ffe25640 1 -- 172.21.15.122:0/424217455 shutdown_connections 2024-03-25T18:25:24.068 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.063+0000 7ff8ffe25640 1 -- 172.21.15.122:0/424217455 >> 172.21.15.122:0/424217455 conn(0x7ff8f80b7ea0 msgr2=0x7ff8f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:24.068 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.063+0000 7ff8ffe25640 1 -- 172.21.15.122:0/424217455 shutdown_connections 2024-03-25T18:25:24.068 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.063+0000 7ff8ffe25640 1 -- 172.21.15.122:0/424217455 wait complete. 2024-03-25T18:25:24.068 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.063+0000 7ff8ffe25640 10 monclient: build_initial_monmap 2024-03-25T18:25:24.068 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.063+0000 7ff8ffe25640 10 monclient: monmap: 2024-03-25T18:25:24.069 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:25:24.069 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:24.069 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:25:24.069 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:25:24.069 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:24.069 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:25:24.069 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:25:24.069 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:25:24.069 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:25:24.069 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:24.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.064+0000 7ff8ffe25640 1 librados: starting msgr at 2024-03-25T18:25:24.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.064+0000 7ff8ffe25640 1 librados: starting objecter 2024-03-25T18:25:24.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.064+0000 7ff8ffe25640 1 Processor -- start 2024-03-25T18:25:24.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.064+0000 7ff8ffe25640 1 -- start start 2024-03-25T18:25:24.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.064+0000 7ff8ffe25640 1 librados: setting wanted keys 2024-03-25T18:25:24.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.064+0000 7ff8ffe25640 1 librados: calling monclient init 2024-03-25T18:25:24.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.064+0000 7ff8ffe25640 10 monclient: init 2024-03-25T18:25:24.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.065+0000 7ff8ffe25640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:24.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.065+0000 7ff8ffe25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:24.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.065+0000 7ff8ffe25640 10 monclient(hunting): picked mon.noname-b con 0x7ff8f80a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:25:24.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.065+0000 7ff8ffe25640 10 monclient(hunting): picked mon.noname-a con 0x7ff8f813f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:25:24.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.065+0000 7ff8ffe25640 10 monclient(hunting): picked mon.noname-c con 0x7ff8f81430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:25:24.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.065+0000 7ff8ffe25640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f80abd30 con 0x7ff8f813f710 2024-03-25T18:25:24.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.065+0000 7ff8ffe25640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f80abfa0 con 0x7ff8f80a98f0 2024-03-25T18:25:24.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.065+0000 7ff8ffe25640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8f81d77a0 con 0x7ff8f81430c0 2024-03-25T18:25:24.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.065+0000 7ff8ffe25640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:24.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.065+0000 7ff8ffe25640 10 monclient(hunting): authenticate will time out at 7847.331578s 2024-03-25T18:25:24.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.079+0000 7ff8fd3c0640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ff8f813f710 0x7ff8f81d4820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:60598/0 (socket says 172.21.15.122:60598) 2024-03-25T18:25:24.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.079+0000 7ff8fd3c0640 1 -- 172.21.15.122:0/3171392595 learned_addr learned my addr 172.21.15.122:0/3171392595 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:25:24.082 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.081+0000 7ff8ee7fc640 10 client.?.objecter ms_handle_connect 0x7ff8f813f710 2024-03-25T18:25:24.083 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.081+0000 7ff8ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:24.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.083+0000 7ff8ee7fc640 10 client.?.objecter ms_handle_connect 0x7ff8f80a98f0 2024-03-25T18:25:24.084 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.083+0000 7ff8ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:24.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.091+0000 7ff8ee7fc640 10 client.?.objecter ms_handle_connect 0x7ff8f81430c0 2024-03-25T18:25:24.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.091+0000 7ff8ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:24.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.096+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505789809 0 0) 0x7ff8f80abfa0 con 0x7ff8f80a98f0 2024-03-25T18:25:24.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.096+0000 7ff8ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:24.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.096+0000 7ff8ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:24.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.096+0000 7ff8ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:24.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.096+0000 7ff8ee7fc640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:25:24.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.096+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8dc004eb0 con 0x7ff8f80a98f0 2024-03-25T18:25:24.104 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.103+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988766166 0 0) 0x7ff8f80abd30 con 0x7ff8f813f710 2024-03-25T18:25:24.105 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.103+0000 7ff8ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:24.105 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.103+0000 7ff8ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:24.105 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.103+0000 7ff8ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:24.105 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.103+0000 7ff8ee7fc640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:25:24.105 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.103+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f80abfa0 con 0x7ff8f813f710 2024-03-25T18:25:24.108 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.107+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306516667 0 0) 0x7ff8f81d77a0 con 0x7ff8f81430c0 2024-03-25T18:25:24.108 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.107+0000 7ff8ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:24.108 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.107+0000 7ff8ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:24.108 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.107+0000 7ff8ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:24.108 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.107+0000 7ff8ee7fc640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:25:24.108 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.107+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8f80abd30 con 0x7ff8f81430c0 2024-03-25T18:25:24.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.119+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 796825943 0 0) 0x7ff8dc004eb0 con 0x7ff8f80a98f0 2024-03-25T18:25:24.120 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.119+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8f81d77a0 con 0x7ff8f80a98f0 2024-03-25T18:25:24.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.119+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff8f4001b00 con 0x7ff8f80a98f0 2024-03-25T18:25:24.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.119+0000 7ff8ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:24.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.119+0000 7ff8ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:24.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.119+0000 7ff8ee7fc640 10 monclient(hunting): dump: 2024-03-25T18:25:24.121 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:24.121 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:24.121 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:24.121 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:24.121 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:24.121 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:24.121 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:24.121 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:24.121 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:24.121 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:24.122 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.119+0000 7ff8ee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:24.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.131+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1848794450 0 0) 0x7ff8f80abfa0 con 0x7ff8f813f710 2024-03-25T18:25:24.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.131+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8dc004eb0 con 0x7ff8f813f710 2024-03-25T18:25:24.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.131+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff8e80019f0 con 0x7ff8f813f710 2024-03-25T18:25:24.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.131+0000 7ff8ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:24.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.131+0000 7ff8ee7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:24.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.131+0000 7ff8ee7fc640 10 monclient(hunting): dump: 2024-03-25T18:25:24.133 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:24.133 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:24.133 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:24.133 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:24.133 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:24.133 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:24.133 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:24.133 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:24.133 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:24.133 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:24.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.137+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 891853814 0 0) 0x7ff8f80abd30 con 0x7ff8f81430c0 2024-03-25T18:25:24.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.137+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8f80abfa0 con 0x7ff8f81430c0 2024-03-25T18:25:24.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.137+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff8f0001bd0 con 0x7ff8f81430c0 2024-03-25T18:25:24.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.137+0000 7ff8ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:24.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.137+0000 7ff8ee7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:24.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.137+0000 7ff8ee7fc640 10 monclient(hunting): dump: 2024-03-25T18:25:24.139 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:24.139 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:24.139 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:24.139 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:24.139 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:24.139 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:24.139 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:24.139 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:24.139 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:24.139 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:24.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3429950910 0 0) 0x7ff8f80abfa0 con 0x7ff8f81430c0 2024-03-25T18:25:24.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 >> v1:172.21.15.204:6789/0 conn(0x7ff8f80a98f0 legacy=0x7ff8f81d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 >> v1:172.21.15.122:6789/0 conn(0x7ff8f813f710 legacy=0x7ff8f81d4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:24.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ee7fc640 1 monclient: found mon.c 2024-03-25T18:25:24.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:24.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ee7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:24.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f81d83d0 con 0x7ff8f81430c0 2024-03-25T18:25:24.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ee7fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:24.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:24.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:24.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ffe25640 5 monclient: authenticate success, global_id 4595 2024-03-25T18:25:24.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ffe25640 10 monclient: _renew_subs 2024-03-25T18:25:24.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ffe25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:24.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ffe25640 1 -- 172.21.15.122:0/3171392595 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f81d41e0 con 0x7ff8f81430c0 2024-03-25T18:25:24.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ffe25640 10 client.4595.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:24.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ffe25640 10 monclient: _renew_subs 2024-03-25T18:25:24.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ffe25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:24.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ffe25640 1 -- 172.21.15.122:0/3171392595 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f81d5110 con 0x7ff8f81430c0 2024-03-25T18:25:24.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.148+0000 7ff8ffe25640 1 librados: init done 2024-03-25T18:25:24.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.149+0000 7ff8ffe25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:24.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.149+0000 7ff8ffe25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:24.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.149+0000 7ff8ffe25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:24.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.149+0000 7ff8ffe25640 1 -- 172.21.15.122:0/3171392595 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f81d7930 con 0x7ff8f81430c0 2024-03-25T18:25:24.473 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.472+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff8f0002020 con 0x7ff8f81430c0 2024-03-25T18:25:24.474 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.472+0000 7ff8ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:24.474 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.472+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ff8f0004d00 con 0x7ff8f81430c0 2024-03-25T18:25:24.474 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.472+0000 7ff8ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:24.474 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.472+0000 7ff8ee7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:24.474 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.472+0000 7ff8ee7fc640 10 monclient: dump: 2024-03-25T18:25:24.474 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:24.474 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:24.474 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:24.474 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:24.474 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:24.474 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:24.475 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:24.475 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:24.475 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:24.475 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:24.475 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.472+0000 7ff8effff640 4 set_mon_vals no callback set 2024-03-25T18:25:24.475 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.473+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7ff8f001acd0 con 0x7ff8f81430c0 2024-03-25T18:25:24.476 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.474+0000 7ff8fdbc1640 10 monclient: get_auth_request con 0x7ff8dc06e160 auth_method 0 2024-03-25T18:25:24.476 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.474+0000 7ff8ee7fc640 10 client.4595.objecter ms_handle_connect 0x7ff8dc06e160 2024-03-25T18:25:24.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.479+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7ff8f00549a0 con 0x7ff8f81430c0 2024-03-25T18:25:24.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.479+0000 7ff8ee7fc640 10 client.4595.objecter ms_dispatch 0x7ff8f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:24.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.479+0000 7ff8ee7fc640 3 client.4595.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:24.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.479+0000 7ff8ee7fc640 3 client.4595.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:24.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.479+0000 7ff8ee7fc640 20 client.4595.objecter dump_active .. 0 homeless 2024-03-25T18:25:24.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.479+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff8f00502e0 con 0x7ff8f81430c0 2024-03-25T18:25:24.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.479+0000 7ff8ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:24.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.479+0000 7ff8ee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:24.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.737+0000 7ff8ffe25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:24.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.737+0000 7ff8ffe25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:24.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.737+0000 7ff8ffe25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:24.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.737+0000 7ff8ffe25640 1 -- 172.21.15.122:0/3171392595 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8f8136a20 con 0x7ff8f81430c0 2024-03-25T18:25:24.890 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.888+0000 7ff8ee7fc640 1 -- 172.21.15.122:0/3171392595 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7ff8f00523e0 con 0x7ff8f81430c0 2024-03-25T18:25:24.890 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.889+0000 7ff8ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:24.890 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.889+0000 7ff8ee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:24.890 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:25:24.891 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:24.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.891+0000 7ff8ffe25640 10 librados: watch_flush enter 2024-03-25T18:25:24.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.892+0000 7ff8ffe25640 10 librados: watch_flush exit 2024-03-25T18:25:24.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.892+0000 7ff8ffe25640 20 client.4595.objecter shutdown clearing up homeless session... 2024-03-25T18:25:24.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.892+0000 7ff8ffe25640 10 client.4595.objecter successfully canceled tick 2024-03-25T18:25:24.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.892+0000 7ff8ffe25640 1 -- 172.21.15.122:0/3171392595 >> v1:172.21.15.204:6800/676228005 conn(0x7ff8dc06e160 legacy=0x7ff8dc070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.892+0000 7ff8ffe25640 10 monclient: shutdown 2024-03-25T18:25:24.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.892+0000 7ff8ffe25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:24.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.892+0000 7ff8ffe25640 1 -- 172.21.15.122:0/3171392595 >> v1:172.21.15.122:6790/0 conn(0x7ff8f81430c0 legacy=0x7ff8f81385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:24.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.892+0000 7ff8ffe25640 1 -- 172.21.15.122:0/3171392595 shutdown_connections 2024-03-25T18:25:24.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.892+0000 7ff8ffe25640 1 -- 172.21.15.122:0/3171392595 >> 172.21.15.122:0/3171392595 conn(0x7ff8f80b7ea0 msgr2=0x7ff8f813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:24.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.892+0000 7ff8ffe25640 1 -- 172.21.15.122:0/3171392595 shutdown_connections 2024-03-25T18:25:24.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.892+0000 7ff8ffe25640 1 -- 172.21.15.122:0/3171392595 wait complete. 2024-03-25T18:25:24.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:24.892+0000 7ff8ffe25640 1 librados: shutdown 2024-03-25T18:25:24.905 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:31.711 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:31.824 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.823+0000 7feecf69a640 10 monclient: get_monmap_and_config 2024-03-25T18:25:31.824 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.823+0000 7feecf69a640 10 monclient: build_initial_monmap 2024-03-25T18:25:31.824 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.823+0000 7feecf69a640 10 monclient: monmap: 2024-03-25T18:25:31.824 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:25:31.825 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:31.825 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:25:31.824327+0000 2024-03-25T18:25:31.825 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:25:31.824327+0000 2024-03-25T18:25:31.825 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:31.825 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:31.825 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:25:31.825 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:25:31.825 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:25:31.825 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:31.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.823+0000 7feecf69a640 1 Processor -- start 2024-03-25T18:25:31.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.824+0000 7feecf69a640 1 -- start start 2024-03-25T18:25:31.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.824+0000 7feecf69a640 10 monclient: init 2024-03-25T18:25:31.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.824+0000 7feecf69a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:31.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.824+0000 7feecf69a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:31.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.824+0000 7feecf69a640 10 monclient(hunting): picked mon.noname-b con 0x7feec80a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:25:31.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.824+0000 7feecf69a640 10 monclient(hunting): picked mon.noname-a con 0x7feec813f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:25:31.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.824+0000 7feecf69a640 10 monclient(hunting): picked mon.noname-c con 0x7feec81430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:25:31.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.824+0000 7feecf69a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec8142520 con 0x7feec813f710 2024-03-25T18:25:31.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.824+0000 7feecf69a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec80b28a0 con 0x7feec81430c0 2024-03-25T18:25:31.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.824+0000 7feecf69a640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec80b2a30 con 0x7feec80a98f0 2024-03-25T18:25:31.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.824+0000 7feecf69a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:31.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.824+0000 7feecf69a640 10 monclient(hunting): authenticate will time out at 7855.090899s 2024-03-25T18:25:31.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.833+0000 7feecdc37640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7feec81430c0 0x7feec814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:34574/0 (socket says 172.21.15.122:34574) 2024-03-25T18:25:31.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.833+0000 7feecdc37640 1 -- 172.21.15.122:0/1327222816 learned_addr learned my addr 172.21.15.122:0/1327222816 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:25:31.851 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.850+0000 7feebffff640 1 -- 172.21.15.122:0/1327222816 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137622040 0 0) 0x7feec80b2a30 con 0x7feec80a98f0 2024-03-25T18:25:31.851 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.850+0000 7feebffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.851 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.850+0000 7feebffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.851 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.850+0000 7feebffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.851 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.850+0000 7feebffff640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:25:31.851 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.850+0000 7feebffff640 1 -- 172.21.15.122:0/1327222816 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeb4003520 con 0x7feec80a98f0 2024-03-25T18:25:31.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.850+0000 7feebffff640 1 -- 172.21.15.122:0/1327222816 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755203865 0 0) 0x7feec8142520 con 0x7feec813f710 2024-03-25T18:25:31.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.850+0000 7feebffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.850+0000 7feebffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.850+0000 7feebffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.850+0000 7feebffff640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:25:31.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.850+0000 7feebffff640 1 -- 172.21.15.122:0/1327222816 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feec80b2a30 con 0x7feec813f710 2024-03-25T18:25:31.857 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.856+0000 7feebffff640 1 -- 172.21.15.122:0/1327222816 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617746829 0 0) 0x7feec80b28a0 con 0x7feec81430c0 2024-03-25T18:25:31.857 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.856+0000 7feebffff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.856+0000 7feebffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.856+0000 7feebffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.856+0000 7feebffff640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:25:31.858 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.856+0000 7feebffff640 1 -- 172.21.15.122:0/1327222816 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feec8142520 con 0x7feec81430c0 2024-03-25T18:25:31.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.857+0000 7feebffff640 1 -- 172.21.15.122:0/1327222816 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2697052888 0 0) 0x7feec80b2a30 con 0x7feec813f710 2024-03-25T18:25:31.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.857+0000 7feebffff640 1 -- 172.21.15.122:0/1327222816 >> v1:172.21.15.122:6790/0 conn(0x7feec80a98f0 legacy=0x7feec80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:31.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.857+0000 7feebffff640 1 -- 172.21.15.122:0/1327222816 >> v1:172.21.15.204:6789/0 conn(0x7feec81430c0 legacy=0x7feec814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:31.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.857+0000 7feebffff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:31.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.857+0000 7feebffff640 1 monclient: found mon.noname-a 2024-03-25T18:25:31.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.857+0000 7feebffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:31.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.857+0000 7feebffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:25:31.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.857+0000 7feebffff640 1 -- 172.21.15.122:0/1327222816 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec80b44c0 con 0x7feec813f710 2024-03-25T18:25:31.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.857+0000 7feebffff640 10 monclient: _finish_auth 0 2024-03-25T18:25:31.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.857+0000 7feebffff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:31.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.857+0000 7feebffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:31.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.857+0000 7feecf69a640 5 monclient: authenticate success, global_id 4641 2024-03-25T18:25:31.859 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.857+0000 7feecf69a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:31.860 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.859+0000 7feebffff640 1 -- 172.21.15.122:0/1327222816 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7feeb8001630 con 0x7feec813f710 2024-03-25T18:25:31.860 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.859+0000 7feebffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:31.860 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.859+0000 7feebffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:31.861 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.859+0000 7feebffff640 10 monclient: dump: 2024-03-25T18:25:31.861 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:31.861 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:31.861 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:31.861 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:31.861 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:31.861 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:31.861 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:31.861 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:31.861 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:31.861 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:31.861 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.859+0000 7feecf69a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:31.880 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.879+0000 7feebffff640 1 -- 172.21.15.122:0/1327222816 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feeb8001880 con 0x7feec813f710 2024-03-25T18:25:31.880 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.879+0000 7feebffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:31.880 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.879+0000 7feebffff640 1 -- 172.21.15.122:0/1327222816 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7feeb8001c10 con 0x7feec813f710 2024-03-25T18:25:31.881 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.879+0000 7feecf69a640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:31.881 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.879+0000 7feecf69a640 4 set_mon_vals no callback set 2024-03-25T18:25:31.881 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.879+0000 7feebffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:31.881 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.879+0000 7feebffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:31.881 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.879+0000 7feebffff640 10 monclient: dump: 2024-03-25T18:25:31.881 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:31.881 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:31.881 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:31.881 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:31.881 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:31.881 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:31.881 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:31.881 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:31.881 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:31.882 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:31.883 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.880+0000 7feecf69a640 10 monclient: shutdown 2024-03-25T18:25:31.883 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.880+0000 7feecf69a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:31.883 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.880+0000 7feecf69a640 1 -- 172.21.15.122:0/1327222816 >> v1:172.21.15.122:6789/0 conn(0x7feec813f710 legacy=0x7feec81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:31.883 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.880+0000 7feecf69a640 1 -- 172.21.15.122:0/1327222816 shutdown_connections 2024-03-25T18:25:31.883 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.880+0000 7feecf69a640 1 -- 172.21.15.122:0/1327222816 >> 172.21.15.122:0/1327222816 conn(0x7feec80b7ea0 msgr2=0x7feec80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:31.883 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.880+0000 7feecf69a640 1 -- 172.21.15.122:0/1327222816 shutdown_connections 2024-03-25T18:25:31.883 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.880+0000 7feecf69a640 1 -- 172.21.15.122:0/1327222816 wait complete. 2024-03-25T18:25:31.883 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.881+0000 7feecf69a640 10 monclient: build_initial_monmap 2024-03-25T18:25:31.883 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.881+0000 7feecf69a640 10 monclient: monmap: 2024-03-25T18:25:31.884 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:25:31.884 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:31.884 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:25:31.884 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:25:31.884 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:31.884 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:25:31.884 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:25:31.884 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:25:31.884 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:25:31.884 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:31.884 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.881+0000 7feecf69a640 1 librados: starting msgr at 2024-03-25T18:25:31.884 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.881+0000 7feecf69a640 1 librados: starting objecter 2024-03-25T18:25:31.884 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.881+0000 7feecf69a640 1 Processor -- start 2024-03-25T18:25:31.886 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.881+0000 7feecf69a640 1 -- start start 2024-03-25T18:25:31.886 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.881+0000 7feecf69a640 1 librados: setting wanted keys 2024-03-25T18:25:31.886 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.881+0000 7feecf69a640 1 librados: calling monclient init 2024-03-25T18:25:31.886 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.881+0000 7feecf69a640 10 monclient: init 2024-03-25T18:25:31.886 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.882+0000 7feecf69a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:31.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.882+0000 7feecf69a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:31.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.882+0000 7feecf69a640 10 monclient(hunting): picked mon.noname-c con 0x7feec80a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:25:31.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.882+0000 7feecf69a640 10 monclient(hunting): picked mon.noname-a con 0x7feec813f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:25:31.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.882+0000 7feecf69a640 10 monclient(hunting): picked mon.noname-b con 0x7feec81430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:25:31.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.882+0000 7feecf69a640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec80abb00 con 0x7feec813f710 2024-03-25T18:25:31.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.882+0000 7feecf69a640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec8138f10 con 0x7feec81430c0 2024-03-25T18:25:31.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.882+0000 7feecf69a640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feec80abd30 con 0x7feec80a98f0 2024-03-25T18:25:31.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.882+0000 7feecf69a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:31.887 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.882+0000 7feecf69a640 10 monclient(hunting): authenticate will time out at 7855.148510s 2024-03-25T18:25:31.896 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.895+0000 7feecdc37640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7feec81430c0 0x7feec81e1fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:34606/0 (socket says 172.21.15.122:34606) 2024-03-25T18:25:31.896 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.895+0000 7feecdc37640 1 -- 172.21.15.122:0/754206810 learned_addr learned my addr 172.21.15.122:0/754206810 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:25:31.897 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.896+0000 7feebdffb640 10 client.?.objecter ms_handle_connect 0x7feec81430c0 2024-03-25T18:25:31.897 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.896+0000 7feebdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:31.901 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.900+0000 7feebdffb640 10 client.?.objecter ms_handle_connect 0x7feec813f710 2024-03-25T18:25:31.901 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.900+0000 7feebdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:31.903 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.902+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156459177 0 0) 0x7feec8138f10 con 0x7feec81430c0 2024-03-25T18:25:31.903 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.902+0000 7feebdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.903 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.902+0000 7feebdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.903 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.902+0000 7feebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.904 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.902+0000 7feebdffb640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:25:31.904 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.902+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feeac004eb0 con 0x7feec81430c0 2024-03-25T18:25:31.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.906+0000 7feebdffb640 10 client.?.objecter ms_handle_connect 0x7feec80a98f0 2024-03-25T18:25:31.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.906+0000 7feebdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:31.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.907+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135747661 0 0) 0x7feec80abb00 con 0x7feec813f710 2024-03-25T18:25:31.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.907+0000 7feebdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.907+0000 7feebdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.907+0000 7feebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.907+0000 7feebdffb640 10 monclient(hunting): my global_id is 4644 2024-03-25T18:25:31.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.907+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feec8138f10 con 0x7feec813f710 2024-03-25T18:25:31.911 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.910+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638737904 0 0) 0x7feec80abd30 con 0x7feec80a98f0 2024-03-25T18:25:31.912 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.910+0000 7feebdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:31.912 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.910+0000 7feebdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:31.912 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.910+0000 7feebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:31.912 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.910+0000 7feebdffb640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:25:31.912 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.910+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feec80abb00 con 0x7feec80a98f0 2024-03-25T18:25:31.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.914+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3837523471 0 0) 0x7feec8138f10 con 0x7feec813f710 2024-03-25T18:25:31.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.914+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feec80abd30 con 0x7feec813f710 2024-03-25T18:25:31.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.914+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7feeb8001b90 con 0x7feec813f710 2024-03-25T18:25:31.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.914+0000 7feebdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:31.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.914+0000 7feebdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:31.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.914+0000 7feebdffb640 10 monclient(hunting): dump: 2024-03-25T18:25:31.916 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:31.916 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:31.916 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:31.916 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:31.917 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:31.917 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:31.917 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:31.917 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:31.917 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:31.917 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:31.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.914+0000 7feebdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:31.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.914+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2529629813 0 0) 0x7feeac004eb0 con 0x7feec81430c0 2024-03-25T18:25:31.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.914+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feec8138f10 con 0x7feec81430c0 2024-03-25T18:25:31.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.914+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7feec0001bd0 con 0x7feec81430c0 2024-03-25T18:25:31.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.914+0000 7feebdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:31.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.914+0000 7feebdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:31.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.914+0000 7feebdffb640 10 monclient(hunting): dump: 2024-03-25T18:25:31.917 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:31.917 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:31.918 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:31.918 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:31.918 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:31.918 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:31.918 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:31.918 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:31.918 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:31.918 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:31.922 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.920+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 295233192 0 0) 0x7feec80abd30 con 0x7feec813f710 2024-03-25T18:25:31.922 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.920+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 >> v1:172.21.15.122:6790/0 conn(0x7feec80a98f0 legacy=0x7feec81d77d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:31.922 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.920+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 >> v1:172.21.15.204:6789/0 conn(0x7feec81430c0 legacy=0x7feec81e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:31.922 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.920+0000 7feebdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:31.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.920+0000 7feebdffb640 1 monclient: found mon.a 2024-03-25T18:25:31.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.920+0000 7feebdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:31.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.920+0000 7feebdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:25:31.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.920+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feec814e7a0 con 0x7feec813f710 2024-03-25T18:25:31.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.920+0000 7feebdffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:31.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.920+0000 7feebdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:31.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.920+0000 7feebdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:31.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.920+0000 7feecf69a640 5 monclient: authenticate success, global_id 4644 2024-03-25T18:25:31.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.920+0000 7feecf69a640 10 monclient: _renew_subs 2024-03-25T18:25:31.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.920+0000 7feecf69a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:25:31.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.920+0000 7feecf69a640 1 -- 172.21.15.122:0/754206810 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feec81380c0 con 0x7feec813f710 2024-03-25T18:25:31.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.921+0000 7feecf69a640 10 client.4644.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:31.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.921+0000 7feecf69a640 10 monclient: _renew_subs 2024-03-25T18:25:31.923 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.921+0000 7feecf69a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:25:31.924 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.921+0000 7feecf69a640 1 -- 172.21.15.122:0/754206810 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feec8138980 con 0x7feec813f710 2024-03-25T18:25:31.924 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.921+0000 7feecf69a640 1 librados: init done 2024-03-25T18:25:31.924 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.922+0000 7feecf69a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:31.924 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.922+0000 7feecf69a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:31.924 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.922+0000 7feecf69a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:25:31.924 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:31.922+0000 7feecf69a640 1 -- 172.21.15.122:0/754206810 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feec8138c80 con 0x7feec813f710 2024-03-25T18:25:32.220 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.218+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feeb80038a0 con 0x7feec813f710 2024-03-25T18:25:32.220 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.218+0000 7feebdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:32.220 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.218+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7feeb8003bb0 con 0x7feec813f710 2024-03-25T18:25:32.220 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.218+0000 7feebdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:32.220 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.218+0000 7feebdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:32.220 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.218+0000 7feebdffb640 10 monclient: dump: 2024-03-25T18:25:32.220 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:32.220 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:32.221 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:32.221 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:32.221 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:32.221 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:32.221 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:32.221 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:32.221 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:32.221 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:32.221 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.218+0000 7feebf7fe640 4 set_mon_vals no callback set 2024-03-25T18:25:32.222 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.219+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7feeb8019b80 con 0x7feec813f710 2024-03-25T18:25:32.222 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.220+0000 7feecd436640 10 monclient: get_auth_request con 0x7feeac06de20 auth_method 0 2024-03-25T18:25:32.222 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.220+0000 7feebdffb640 10 client.4644.objecter ms_handle_connect 0x7feeac06de20 2024-03-25T18:25:32.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.225+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7feeb80536e0 con 0x7feec813f710 2024-03-25T18:25:32.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.225+0000 7feebdffb640 10 client.4644.objecter ms_dispatch 0x7feec8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:32.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.225+0000 7feebdffb640 3 client.4644.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:32.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.225+0000 7feebdffb640 3 client.4644.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:32.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.225+0000 7feebdffb640 20 client.4644.objecter dump_active .. 0 homeless 2024-03-25T18:25:32.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.225+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feeb804f1f0 con 0x7feec813f710 2024-03-25T18:25:32.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.225+0000 7feebdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:32.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.225+0000 7feebdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:32.318 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:09:26.060 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:32.372 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:09:26.114 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:32.402 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:09:26.144 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:32.417 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:09:26.163 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:32.423 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:09:26.168 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:32.448 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:09:26.190 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:32.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.449+0000 7feecf69a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:32.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.449+0000 7feecf69a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:32.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.449+0000 7feecf69a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:25:32.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.449+0000 7feecf69a640 1 -- 172.21.15.122:0/754206810 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feec8074cc0 con 0x7feec813f710 2024-03-25T18:25:32.463 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:09:26.206 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:32.465 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:09:26.210 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:32.496 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:09:26.239 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:32.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.496+0000 7feebdffb640 1 -- 172.21.15.122:0/754206810 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7feeb8051120 con 0x7feec813f710 2024-03-25T18:25:32.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.496+0000 7feebdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:32.498 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.496+0000 7feebdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:32.498 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:25:32.498 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:32.501 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.499+0000 7feecf69a640 10 librados: watch_flush enter 2024-03-25T18:25:32.501 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.499+0000 7feecf69a640 10 librados: watch_flush exit 2024-03-25T18:25:32.501 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.499+0000 7feecf69a640 20 client.4644.objecter shutdown clearing up homeless session... 2024-03-25T18:25:32.501 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.499+0000 7feecf69a640 10 client.4644.objecter successfully canceled tick 2024-03-25T18:25:32.501 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.499+0000 7feecf69a640 1 -- 172.21.15.122:0/754206810 >> v1:172.21.15.204:6800/676228005 conn(0x7feeac06de20 legacy=0x7feeac0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.501 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.499+0000 7feecf69a640 10 monclient: shutdown 2024-03-25T18:25:32.501 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.499+0000 7feecf69a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:32.501 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.499+0000 7feecf69a640 1 -- 172.21.15.122:0/754206810 >> v1:172.21.15.122:6789/0 conn(0x7feec813f710 legacy=0x7feec81d7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:32.501 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.499+0000 7feecf69a640 1 -- 172.21.15.122:0/754206810 shutdown_connections 2024-03-25T18:25:32.502 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.499+0000 7feecf69a640 1 -- 172.21.15.122:0/754206810 >> 172.21.15.122:0/754206810 conn(0x7feec80b7ea0 msgr2=0x7feec813d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:32.502 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.499+0000 7feecf69a640 1 -- 172.21.15.122:0/754206810 shutdown_connections 2024-03-25T18:25:32.502 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.499+0000 7feecf69a640 1 -- 172.21.15.122:0/754206810 wait complete. 2024-03-25T18:25:32.502 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:32.499+0000 7feecf69a640 1 librados: shutdown 2024-03-25T18:25:32.512 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:39.319 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:39.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.434+0000 7fae0ebaf640 10 monclient: get_monmap_and_config 2024-03-25T18:25:39.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.434+0000 7fae0ebaf640 10 monclient: build_initial_monmap 2024-03-25T18:25:39.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.434+0000 7fae0ebaf640 10 monclient: monmap: 2024-03-25T18:25:39.436 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:25:39.436 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:39.436 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:25:39.435324+0000 2024-03-25T18:25:39.436 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:25:39.435324+0000 2024-03-25T18:25:39.436 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:39.436 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:39.436 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:25:39.436 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:25:39.436 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:25:39.436 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:39.437 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.435+0000 7fae0ebaf640 1 Processor -- start 2024-03-25T18:25:39.437 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.435+0000 7fae0ebaf640 1 -- start start 2024-03-25T18:25:39.437 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.435+0000 7fae0ebaf640 10 monclient: init 2024-03-25T18:25:39.437 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.435+0000 7fae0ebaf640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:39.437 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.435+0000 7fae0ebaf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:39.437 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.436+0000 7fae0ebaf640 10 monclient(hunting): picked mon.noname-a con 0x7fae080a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:25:39.437 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.436+0000 7fae0ebaf640 10 monclient(hunting): picked mon.noname-c con 0x7fae0813f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:25:39.437 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.436+0000 7fae0ebaf640 10 monclient(hunting): picked mon.noname-b con 0x7fae081430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:25:39.437 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.436+0000 7fae0ebaf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae08142520 con 0x7fae080a98f0 2024-03-25T18:25:39.437 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.436+0000 7fae0ebaf640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae080b28a0 con 0x7fae0813f710 2024-03-25T18:25:39.438 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.436+0000 7fae0ebaf640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae080b2a30 con 0x7fae081430c0 2024-03-25T18:25:39.438 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.436+0000 7fae0ebaf640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:39.438 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.436+0000 7fae0ebaf640 10 monclient(hunting): authenticate will time out at 7862.702070s 2024-03-25T18:25:39.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.448+0000 7fae0d14c640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7fae081430c0 0x7fae0814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:41944/0 (socket says 172.21.15.122:41944) 2024-03-25T18:25:39.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.448+0000 7fae0d14c640 1 -- 172.21.15.122:0/3213941874 learned_addr learned my addr 172.21.15.122:0/3213941874 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:25:39.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.458+0000 7fae077fe640 1 -- 172.21.15.122:0/3213941874 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552277962 0 0) 0x7fae080b28a0 con 0x7fae0813f710 2024-03-25T18:25:39.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.458+0000 7fae077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.458+0000 7fae077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.458+0000 7fae077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.458+0000 7fae077fe640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:25:39.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.458+0000 7fae077fe640 1 -- 172.21.15.122:0/3213941874 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadf8003520 con 0x7fae0813f710 2024-03-25T18:25:39.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.459+0000 7fae077fe640 1 -- 172.21.15.122:0/3213941874 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3486018413 0 0) 0x7fae080b2a30 con 0x7fae081430c0 2024-03-25T18:25:39.461 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.459+0000 7fae077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.461 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.459+0000 7fae077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.461 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.459+0000 7fae077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.461 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.459+0000 7fae077fe640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:25:39.461 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.459+0000 7fae077fe640 1 -- 172.21.15.122:0/3213941874 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae080b28a0 con 0x7fae081430c0 2024-03-25T18:25:39.467 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.465+0000 7fae077fe640 1 -- 172.21.15.122:0/3213941874 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1558445389 0 0) 0x7fadf8003520 con 0x7fae0813f710 2024-03-25T18:25:39.467 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.465+0000 7fae077fe640 1 -- 172.21.15.122:0/3213941874 >> v1:172.21.15.122:6790/0 conn(0x7fae081430c0 legacy=0x7fae0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.467 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.465+0000 7fae077fe640 1 -- 172.21.15.122:0/3213941874 >> v1:172.21.15.122:6789/0 conn(0x7fae080a98f0 legacy=0x7fae080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.467 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.465+0000 7fae077fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:39.467 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.465+0000 7fae077fe640 1 monclient: found mon.noname-c 2024-03-25T18:25:39.467 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.465+0000 7fae077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:39.467 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.465+0000 7fae077fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:25:39.467 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.465+0000 7fae077fe640 1 -- 172.21.15.122:0/3213941874 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae080b44c0 con 0x7fae0813f710 2024-03-25T18:25:39.467 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.465+0000 7fae077fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:39.467 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.465+0000 7fae077fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:39.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.465+0000 7fae077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:39.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.465+0000 7fae077fe640 1 -- 172.21.15.122:0/3213941874 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fadf4001690 con 0x7fae0813f710 2024-03-25T18:25:39.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.465+0000 7fae077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.465+0000 7fae077fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:39.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.465+0000 7fae077fe640 10 monclient: dump: 2024-03-25T18:25:39.468 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:39.468 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:39.468 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:39.468 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:39.468 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.468 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:39.468 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:39.468 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:39.468 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:39.468 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:39.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.465+0000 7fae0ebaf640 5 monclient: authenticate success, global_id 4636 2024-03-25T18:25:39.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.465+0000 7fae0ebaf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:39.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.476+0000 7fae077fe640 1 -- 172.21.15.122:0/3213941874 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fadf40017f0 con 0x7fae0813f710 2024-03-25T18:25:39.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.476+0000 7fae077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:39.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.476+0000 7fae077fe640 1 -- 172.21.15.122:0/3213941874 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fadf4001c00 con 0x7fae0813f710 2024-03-25T18:25:39.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.476+0000 7fae077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.477 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.476+0000 7fae077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:39.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.476+0000 7fae077fe640 10 monclient: dump: 2024-03-25T18:25:39.478 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:39.478 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:39.478 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:39.478 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:39.478 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.478 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:39.478 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:39.478 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:39.478 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:39.478 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:39.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.476+0000 7fae0ebaf640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:39.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.476+0000 7fae0ebaf640 4 set_mon_vals no callback set 2024-03-25T18:25:39.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.477+0000 7fae0ebaf640 10 monclient: shutdown 2024-03-25T18:25:39.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.477+0000 7fae0ebaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:39.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.477+0000 7fae0ebaf640 1 -- 172.21.15.122:0/3213941874 >> v1:172.21.15.204:6789/0 conn(0x7fae0813f710 legacy=0x7fae081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.477+0000 7fae0ebaf640 1 -- 172.21.15.122:0/3213941874 shutdown_connections 2024-03-25T18:25:39.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.477+0000 7fae0ebaf640 1 -- 172.21.15.122:0/3213941874 >> 172.21.15.122:0/3213941874 conn(0x7fae080b7ea0 msgr2=0x7fae080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:39.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.477+0000 7fae0ebaf640 1 -- 172.21.15.122:0/3213941874 shutdown_connections 2024-03-25T18:25:39.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.477+0000 7fae0ebaf640 1 -- 172.21.15.122:0/3213941874 wait complete. 2024-03-25T18:25:39.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.477+0000 7fae0ebaf640 10 monclient: build_initial_monmap 2024-03-25T18:25:39.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.477+0000 7fae0ebaf640 10 monclient: monmap: 2024-03-25T18:25:39.481 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:25:39.481 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:39.481 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:25:39.481 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:25:39.481 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:39.481 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:25:39.482 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:25:39.482 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:25:39.482 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:25:39.482 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:39.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.478+0000 7fae0ebaf640 1 librados: starting msgr at 2024-03-25T18:25:39.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.478+0000 7fae0ebaf640 1 librados: starting objecter 2024-03-25T18:25:39.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.478+0000 7fae0ebaf640 1 Processor -- start 2024-03-25T18:25:39.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.478+0000 7fae0ebaf640 1 -- start start 2024-03-25T18:25:39.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.478+0000 7fae0ebaf640 1 librados: setting wanted keys 2024-03-25T18:25:39.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.478+0000 7fae0ebaf640 1 librados: calling monclient init 2024-03-25T18:25:39.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.478+0000 7fae0ebaf640 10 monclient: init 2024-03-25T18:25:39.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.479+0000 7fae0ebaf640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:39.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.479+0000 7fae0ebaf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:39.482 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.479+0000 7fae0ebaf640 10 monclient(hunting): picked mon.noname-a con 0x7fae080a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:25:39.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.479+0000 7fae0ebaf640 10 monclient(hunting): picked mon.noname-b con 0x7fae0813f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:25:39.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.479+0000 7fae0ebaf640 10 monclient(hunting): picked mon.noname-c con 0x7fae081430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:25:39.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.479+0000 7fae0ebaf640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae08138c10 con 0x7fae080a98f0 2024-03-25T18:25:39.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.479+0000 7fae0ebaf640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae08138e60 con 0x7fae0813f710 2024-03-25T18:25:39.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.479+0000 7fae0ebaf640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae08139140 con 0x7fae081430c0 2024-03-25T18:25:39.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.479+0000 7fae0ebaf640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:39.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.479+0000 7fae0ebaf640 10 monclient(hunting): authenticate will time out at 7862.745225s 2024-03-25T18:25:39.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.493+0000 7fae07fff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fae0813f710 0x7fae08132440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:39126/0 (socket says 172.21.15.122:39126) 2024-03-25T18:25:39.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.493+0000 7fae07fff640 1 -- 172.21.15.122:0/1998537681 learned_addr learned my addr 172.21.15.122:0/1998537681 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:25:39.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.496+0000 7fae057fa640 10 client.?.objecter ms_handle_connect 0x7fae0813f710 2024-03-25T18:25:39.497 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.496+0000 7fae057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:39.502 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.500+0000 7fae057fa640 10 client.?.objecter ms_handle_connect 0x7fae080a98f0 2024-03-25T18:25:39.502 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.500+0000 7fae057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:39.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.503+0000 7fae057fa640 10 client.?.objecter ms_handle_connect 0x7fae081430c0 2024-03-25T18:25:39.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.504+0000 7fae057fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:39.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.512+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430713595 0 0) 0x7fae08138e60 con 0x7fae0813f710 2024-03-25T18:25:39.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.512+0000 7fae057fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.512+0000 7fae057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.512+0000 7fae057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.512+0000 7fae057fa640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:25:39.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.512+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadec004eb0 con 0x7fae0813f710 2024-03-25T18:25:39.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.514+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2710047429 0 0) 0x7fae08139140 con 0x7fae081430c0 2024-03-25T18:25:39.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.514+0000 7fae057fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.514+0000 7fae057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.514+0000 7fae057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.514+0000 7fae057fa640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:25:39.516 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.514+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae08138e60 con 0x7fae081430c0 2024-03-25T18:25:39.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.522+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814142168 0 0) 0x7fae08138c10 con 0x7fae080a98f0 2024-03-25T18:25:39.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.522+0000 7fae057fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:39.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.522+0000 7fae057fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:39.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.522+0000 7fae057fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:39.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.522+0000 7fae057fa640 10 monclient(hunting): my global_id is 4650 2024-03-25T18:25:39.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.522+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae08139140 con 0x7fae080a98f0 2024-03-25T18:25:39.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.534+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 775677971 0 0) 0x7fae08139140 con 0x7fae080a98f0 2024-03-25T18:25:39.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.534+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae08138c10 con 0x7fae080a98f0 2024-03-25T18:25:39.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.534+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fadfc001860 con 0x7fae080a98f0 2024-03-25T18:25:39.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.534+0000 7fae057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.534+0000 7fae057fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:39.536 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.534+0000 7fae057fa640 10 monclient(hunting): dump: 2024-03-25T18:25:39.536 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:39.536 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:39.536 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:39.537 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:39.537 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.537 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:39.537 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:39.537 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:39.537 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:39.537 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:39.537 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.534+0000 7fae057fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:39.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.537+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 820065758 0 0) 0x7fadec004eb0 con 0x7fae0813f710 2024-03-25T18:25:39.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.537+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae08139140 con 0x7fae0813f710 2024-03-25T18:25:39.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.537+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fadf4001690 con 0x7fae0813f710 2024-03-25T18:25:39.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.537+0000 7fae057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.537+0000 7fae057fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:39.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.537+0000 7fae057fa640 10 monclient(hunting): dump: 2024-03-25T18:25:39.539 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:39.539 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:39.539 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:39.539 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:39.539 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.539 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:39.539 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:39.540 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:39.540 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:39.540 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:39.543 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.542+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2331911645 0 0) 0x7fae08138e60 con 0x7fae081430c0 2024-03-25T18:25:39.543 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.542+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadec004eb0 con 0x7fae081430c0 2024-03-25T18:25:39.543 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.542+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fae00001b90 con 0x7fae081430c0 2024-03-25T18:25:39.543 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.542+0000 7fae057fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.543 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.542+0000 7fae057fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:39.544 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.542+0000 7fae057fa640 10 monclient(hunting): dump: 2024-03-25T18:25:39.544 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:39.544 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:39.544 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:39.544 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:39.544 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.544 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:39.544 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:39.544 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:39.544 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:39.544 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:39.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2295100584 0 0) 0x7fadec004eb0 con 0x7fae081430c0 2024-03-25T18:25:39.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 >> v1:172.21.15.204:6789/0 conn(0x7fae0813f710 legacy=0x7fae08132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 >> v1:172.21.15.122:6789/0 conn(0x7fae080a98f0 legacy=0x7fae08137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:39.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae057fa640 10 monclient: _finish_hunting 0 2024-03-25T18:25:39.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae057fa640 1 monclient: found mon.c 2024-03-25T18:25:39.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae057fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:39.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae057fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:39.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae08139920 con 0x7fae081430c0 2024-03-25T18:25:39.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae057fa640 10 monclient: _finish_auth 0 2024-03-25T18:25:39.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae057fa640 10 monclient: _check_auth_tickets 2024-03-25T18:25:39.553 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae057fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:39.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae0ebaf640 5 monclient: authenticate success, global_id 4607 2024-03-25T18:25:39.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae0ebaf640 10 monclient: _renew_subs 2024-03-25T18:25:39.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae0ebaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:39.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae0ebaf640 1 -- 172.21.15.122:0/1998537681 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae081382e0 con 0x7fae081430c0 2024-03-25T18:25:39.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae0ebaf640 10 client.4607.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:39.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae0ebaf640 10 monclient: _renew_subs 2024-03-25T18:25:39.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae0ebaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:39.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae0ebaf640 1 -- 172.21.15.122:0/1998537681 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae080ab8b0 con 0x7fae081430c0 2024-03-25T18:25:39.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.551+0000 7fae0ebaf640 1 librados: init done 2024-03-25T18:25:39.555 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.552+0000 7fae0ebaf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:39.555 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.552+0000 7fae0ebaf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:39.555 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.552+0000 7fae0ebaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:39.555 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.552+0000 7fae0ebaf640 1 -- 172.21.15.122:0/1998537681 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae08137a20 con 0x7fae081430c0 2024-03-25T18:25:39.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.584+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae00002020 con 0x7fae081430c0 2024-03-25T18:25:39.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.584+0000 7fae057fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:39.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.584+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fae00004d00 con 0x7fae081430c0 2024-03-25T18:25:39.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.584+0000 7fae057fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:39.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.584+0000 7fae057fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:39.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.584+0000 7fae057fa640 10 monclient: dump: 2024-03-25T18:25:39.586 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:39.586 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:39.586 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:39.586 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:39.586 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:39.586 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:39.586 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:39.586 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:39.586 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:39.586 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:39.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.584+0000 7fae06ffd640 4 set_mon_vals no callback set 2024-03-25T18:25:39.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.585+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fae0001ad10 con 0x7fae081430c0 2024-03-25T18:25:39.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.585+0000 7fae0c94b640 10 monclient: get_auth_request con 0x7fadec06e110 auth_method 0 2024-03-25T18:25:39.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.586+0000 7fae057fa640 10 client.4607.objecter ms_handle_connect 0x7fadec06e110 2024-03-25T18:25:39.861 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.859+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fae00054a40 con 0x7fae081430c0 2024-03-25T18:25:39.861 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.859+0000 7fae057fa640 10 client.4607.objecter ms_dispatch 0x7fae08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:39.861 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.859+0000 7fae057fa640 3 client.4607.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:39.861 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.859+0000 7fae057fa640 3 client.4607.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:39.861 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.859+0000 7fae057fa640 20 client.4607.objecter dump_active .. 0 homeless 2024-03-25T18:25:39.861 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.859+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae00050380 con 0x7fae081430c0 2024-03-25T18:25:39.861 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.859+0000 7fae057fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:39.861 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:39.859+0000 7fae057fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:40.113 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.112+0000 7fae0ebaf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:40.113 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.112+0000 7fae0ebaf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:40.113 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.112+0000 7fae0ebaf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:40.113 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.112+0000 7fae0ebaf640 1 -- 172.21.15.122:0/1998537681 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fae0813d010 con 0x7fae081430c0 2024-03-25T18:25:40.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.149+0000 7fae057fa640 1 -- 172.21.15.122:0/1998537681 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fae00052480 con 0x7fae081430c0 2024-03-25T18:25:40.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.149+0000 7fae057fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:40.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.149+0000 7fae057fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:40.151 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:25:40.151 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:40.153 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.151+0000 7fae0ebaf640 10 librados: watch_flush enter 2024-03-25T18:25:40.153 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.151+0000 7fae0ebaf640 10 librados: watch_flush exit 2024-03-25T18:25:40.154 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.151+0000 7fae0ebaf640 20 client.4607.objecter shutdown clearing up homeless session... 2024-03-25T18:25:40.154 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.151+0000 7fae0ebaf640 10 client.4607.objecter successfully canceled tick 2024-03-25T18:25:40.154 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.152+0000 7fae0ebaf640 1 -- 172.21.15.122:0/1998537681 >> v1:172.21.15.204:6800/676228005 conn(0x7fadec06e110 legacy=0x7fadec0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:40.154 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.152+0000 7fae0ebaf640 10 monclient: shutdown 2024-03-25T18:25:40.154 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.152+0000 7fae0ebaf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:40.154 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.152+0000 7fae0ebaf640 1 -- 172.21.15.122:0/1998537681 >> v1:172.21.15.122:6790/0 conn(0x7fae081430c0 legacy=0x7fae081e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:40.154 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.152+0000 7fae0ebaf640 1 -- 172.21.15.122:0/1998537681 shutdown_connections 2024-03-25T18:25:40.154 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.152+0000 7fae0ebaf640 1 -- 172.21.15.122:0/1998537681 >> 172.21.15.122:0/1998537681 conn(0x7fae080b7ea0 msgr2=0x7fae08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:40.154 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.152+0000 7fae0ebaf640 1 -- 172.21.15.122:0/1998537681 shutdown_connections 2024-03-25T18:25:40.154 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.152+0000 7fae0ebaf640 1 -- 172.21.15.122:0/1998537681 wait complete. 2024-03-25T18:25:40.154 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:40.152+0000 7fae0ebaf640 1 librados: shutdown 2024-03-25T18:25:40.165 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:46.972 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:47.091 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.089+0000 7f605d4d3640 10 monclient: get_monmap_and_config 2024-03-25T18:25:47.091 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.089+0000 7f605d4d3640 10 monclient: build_initial_monmap 2024-03-25T18:25:47.091 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.089+0000 7f605d4d3640 10 monclient: monmap: 2024-03-25T18:25:47.091 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:25:47.091 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:47.091 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:25:47.090620+0000 2024-03-25T18:25:47.091 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:25:47.090620+0000 2024-03-25T18:25:47.091 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:47.091 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:47.091 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:25:47.091 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:25:47.091 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:25:47.092 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:47.092 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.090+0000 7f605d4d3640 1 Processor -- start 2024-03-25T18:25:47.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.090+0000 7f605d4d3640 1 -- start start 2024-03-25T18:25:47.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.090+0000 7f605d4d3640 10 monclient: init 2024-03-25T18:25:47.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.091+0000 7f605d4d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:47.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.091+0000 7f605d4d3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:47.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.091+0000 7f605d4d3640 10 monclient(hunting): picked mon.noname-b con 0x7f60580a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:25:47.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.091+0000 7f605d4d3640 10 monclient(hunting): picked mon.noname-c con 0x7f605813f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:25:47.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.091+0000 7f605d4d3640 10 monclient(hunting): picked mon.noname-a con 0x7f60581430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:25:47.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.091+0000 7f605d4d3640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6058142520 con 0x7f60581430c0 2024-03-25T18:25:47.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.091+0000 7f605d4d3640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60580b28a0 con 0x7f605813f710 2024-03-25T18:25:47.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.091+0000 7f605d4d3640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60580b2a30 con 0x7f60580a98f0 2024-03-25T18:25:47.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.091+0000 7f605d4d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:47.093 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.091+0000 7f605d4d3640 10 monclient(hunting): authenticate will time out at 7870.357279s 2024-03-25T18:25:47.096 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.095+0000 7f60577fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f60581430c0 0x7f605814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:46216/0 (socket says 172.21.15.122:46216) 2024-03-25T18:25:47.096 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.095+0000 7f60577fe640 1 -- 172.21.15.122:0/3276422508 learned_addr learned my addr 172.21.15.122:0/3276422508 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:25:47.109 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.108+0000 7f6055ffb640 1 -- 172.21.15.122:0/3276422508 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3231298004 0 0) 0x7f6058142520 con 0x7f60581430c0 2024-03-25T18:25:47.110 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.108+0000 7f6055ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:47.110 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.108+0000 7f6055ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:47.110 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.108+0000 7f6055ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:47.110 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.108+0000 7f6055ffb640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:25:47.110 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.108+0000 7f6055ffb640 1 -- 172.21.15.122:0/3276422508 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6040003520 con 0x7f60581430c0 2024-03-25T18:25:47.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.116+0000 7f6055ffb640 1 -- 172.21.15.122:0/3276422508 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3711571661 0 0) 0x7f6040003520 con 0x7f60581430c0 2024-03-25T18:25:47.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.116+0000 7f6055ffb640 1 -- 172.21.15.122:0/3276422508 >> v1:172.21.15.122:6790/0 conn(0x7f60580a98f0 legacy=0x7f60580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.116+0000 7f6055ffb640 1 -- 172.21.15.122:0/3276422508 >> v1:172.21.15.204:6789/0 conn(0x7f605813f710 legacy=0x7f60581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.116+0000 7f6055ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:47.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.116+0000 7f6055ffb640 1 monclient: found mon.noname-a 2024-03-25T18:25:47.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.116+0000 7f6055ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:47.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.116+0000 7f6055ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:25:47.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.116+0000 7f6055ffb640 1 -- 172.21.15.122:0/3276422508 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60580b44c0 con 0x7f60581430c0 2024-03-25T18:25:47.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.116+0000 7f6055ffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:47.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.116+0000 7f6055ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:47.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.116+0000 7f6055ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:47.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.116+0000 7f6055ffb640 1 -- 172.21.15.122:0/3276422508 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f6050001a10 con 0x7f60581430c0 2024-03-25T18:25:47.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.116+0000 7f6055ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:47.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.116+0000 7f6055ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:47.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.116+0000 7f6055ffb640 10 monclient: dump: 2024-03-25T18:25:47.120 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:47.120 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:47.120 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:47.120 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:47.120 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:47.120 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:47.121 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:47.121 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:47.121 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:47.121 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:47.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.116+0000 7f605d4d3640 5 monclient: authenticate success, global_id 4653 2024-03-25T18:25:47.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.116+0000 7f605d4d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:47.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.126+0000 7f6055ffb640 1 -- 172.21.15.122:0/3276422508 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6050001b70 con 0x7f60581430c0 2024-03-25T18:25:47.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.126+0000 7f6055ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:47.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.126+0000 7f6055ffb640 1 -- 172.21.15.122:0/3276422508 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f6050001fc0 con 0x7f60581430c0 2024-03-25T18:25:47.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.126+0000 7f6055ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:47.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.126+0000 7f6055ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:47.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.126+0000 7f6055ffb640 10 monclient: dump: 2024-03-25T18:25:47.127 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:47.127 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:47.128 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:47.128 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:47.128 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:47.128 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:47.128 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:47.128 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:47.128 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:47.128 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:47.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.126+0000 7f605d4d3640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:47.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.126+0000 7f605d4d3640 4 set_mon_vals no callback set 2024-03-25T18:25:47.130 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.127+0000 7f605d4d3640 10 monclient: shutdown 2024-03-25T18:25:47.130 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.127+0000 7f605d4d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:47.130 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.127+0000 7f605d4d3640 1 -- 172.21.15.122:0/3276422508 >> v1:172.21.15.122:6789/0 conn(0x7f60581430c0 legacy=0x7f605814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.130 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.127+0000 7f605d4d3640 1 -- 172.21.15.122:0/3276422508 shutdown_connections 2024-03-25T18:25:47.130 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.127+0000 7f605d4d3640 1 -- 172.21.15.122:0/3276422508 >> 172.21.15.122:0/3276422508 conn(0x7f60580b7ea0 msgr2=0x7f60580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:47.130 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.127+0000 7f605d4d3640 1 -- 172.21.15.122:0/3276422508 shutdown_connections 2024-03-25T18:25:47.130 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.127+0000 7f605d4d3640 1 -- 172.21.15.122:0/3276422508 wait complete. 2024-03-25T18:25:47.130 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.128+0000 7f605d4d3640 10 monclient: build_initial_monmap 2024-03-25T18:25:47.130 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.128+0000 7f605d4d3640 10 monclient: monmap: 2024-03-25T18:25:47.131 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:25:47.131 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:47.131 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:25:47.131 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:25:47.131 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:47.131 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:25:47.131 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:25:47.131 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:25:47.131 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:25:47.131 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:47.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.128+0000 7f605d4d3640 1 librados: starting msgr at 2024-03-25T18:25:47.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.128+0000 7f605d4d3640 1 librados: starting objecter 2024-03-25T18:25:47.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.128+0000 7f605d4d3640 1 Processor -- start 2024-03-25T18:25:47.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.129+0000 7f605d4d3640 1 -- start start 2024-03-25T18:25:47.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.129+0000 7f605d4d3640 1 librados: setting wanted keys 2024-03-25T18:25:47.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.129+0000 7f605d4d3640 1 librados: calling monclient init 2024-03-25T18:25:47.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.129+0000 7f605d4d3640 10 monclient: init 2024-03-25T18:25:47.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.129+0000 7f605d4d3640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:47.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.129+0000 7f605d4d3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:47.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.129+0000 7f605d4d3640 10 monclient(hunting): picked mon.noname-a con 0x7f60580a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:25:47.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.129+0000 7f605d4d3640 10 monclient(hunting): picked mon.noname-b con 0x7f605813f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:25:47.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.129+0000 7f605d4d3640 10 monclient(hunting): picked mon.noname-c con 0x7f60581430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:25:47.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.129+0000 7f605d4d3640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605806fbb0 con 0x7f60580a98f0 2024-03-25T18:25:47.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.129+0000 7f605d4d3640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f605806fe20 con 0x7f605813f710 2024-03-25T18:25:47.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.129+0000 7f605d4d3640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6058070100 con 0x7f60581430c0 2024-03-25T18:25:47.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.130+0000 7f605d4d3640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:47.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.130+0000 7f605d4d3640 10 monclient(hunting): authenticate will time out at 7870.395996s 2024-03-25T18:25:47.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.137+0000 7f6056ffd640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f60580a98f0 0x7f6058070be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:46258/0 (socket says 172.21.15.122:46258) 2024-03-25T18:25:47.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.137+0000 7f6056ffd640 1 -- 172.21.15.122:0/3157486012 learned_addr learned my addr 172.21.15.122:0/3157486012 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:25:47.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.138+0000 7f603f7fe640 10 client.?.objecter ms_handle_connect 0x7f60580a98f0 2024-03-25T18:25:47.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.138+0000 7f603f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:47.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.149+0000 7f603f7fe640 10 client.?.objecter ms_handle_connect 0x7f60581430c0 2024-03-25T18:25:47.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.149+0000 7f603f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:47.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.150+0000 7f603f7fe640 10 client.?.objecter ms_handle_connect 0x7f605813f710 2024-03-25T18:25:47.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.150+0000 7f603f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:47.155 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.154+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810529933 0 0) 0x7f605806fbb0 con 0x7f60580a98f0 2024-03-25T18:25:47.155 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.154+0000 7f603f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:47.155 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.154+0000 7f603f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:47.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.154+0000 7f603f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:47.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.154+0000 7f603f7fe640 10 monclient(hunting): my global_id is 4656 2024-03-25T18:25:47.156 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.154+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6044004eb0 con 0x7f60580a98f0 2024-03-25T18:25:47.157 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.156+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4139876737 0 0) 0x7f605806fe20 con 0x7f605813f710 2024-03-25T18:25:47.158 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.156+0000 7f603f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:47.158 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.156+0000 7f603f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:47.158 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.156+0000 7f603f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:47.158 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.156+0000 7f603f7fe640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:25:47.158 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.156+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f605806fbb0 con 0x7f605813f710 2024-03-25T18:25:47.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.165+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1223151916 0 0) 0x7f6044004eb0 con 0x7f60580a98f0 2024-03-25T18:25:47.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.165+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f605806fe20 con 0x7f60580a98f0 2024-03-25T18:25:47.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.165+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f6048001a30 con 0x7f60580a98f0 2024-03-25T18:25:47.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.165+0000 7f603f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:47.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.165+0000 7f603f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:47.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.165+0000 7f603f7fe640 10 monclient(hunting): dump: 2024-03-25T18:25:47.167 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:47.167 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:47.167 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:47.167 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:47.167 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:47.167 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:47.167 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:47.168 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:47.168 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:47.168 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:47.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.165+0000 7f603f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:47.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.171+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771212650 0 0) 0x7f6058070100 con 0x7f60581430c0 2024-03-25T18:25:47.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.171+0000 7f603f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:47.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.171+0000 7f603f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:47.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.171+0000 7f603f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:47.172 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.171+0000 7f603f7fe640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:25:47.173 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.171+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6044004eb0 con 0x7f60581430c0 2024-03-25T18:25:47.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2355803898 0 0) 0x7f605806fe20 con 0x7f60580a98f0 2024-03-25T18:25:47.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 >> v1:172.21.15.122:6790/0 conn(0x7f60581430c0 legacy=0x7f60581e6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 >> v1:172.21.15.204:6789/0 conn(0x7f605813f710 legacy=0x7f60580712f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f603f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:47.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f603f7fe640 1 monclient: found mon.a 2024-03-25T18:25:47.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f603f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:47.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f603f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:25:47.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60580703e0 con 0x7f60580a98f0 2024-03-25T18:25:47.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f603f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:47.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f603f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:47.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f603f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:47.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f605d4d3640 5 monclient: authenticate success, global_id 4656 2024-03-25T18:25:47.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f605d4d3640 10 monclient: _renew_subs 2024-03-25T18:25:47.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f605d4d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:25:47.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f605d4d3640 1 -- 172.21.15.122:0/3157486012 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f605806f240 con 0x7f60580a98f0 2024-03-25T18:25:47.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f605d4d3640 10 client.4656.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:47.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f605d4d3640 10 monclient: _renew_subs 2024-03-25T18:25:47.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f605d4d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:25:47.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f605d4d3640 1 -- 172.21.15.122:0/3157486012 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f605806e2f0 con 0x7f60580a98f0 2024-03-25T18:25:47.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.181+0000 7f605d4d3640 1 librados: init done 2024-03-25T18:25:47.185 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.182+0000 7f605d4d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:47.185 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.182+0000 7f605d4d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:47.185 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.182+0000 7f605d4d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:25:47.185 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.182+0000 7f605d4d3640 1 -- 172.21.15.122:0/3157486012 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60580627b0 con 0x7f60580a98f0 2024-03-25T18:25:47.487 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.486+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6048001c70 con 0x7f60580a98f0 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.486+0000 7f603f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.486+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f6048003a60 con 0x7f60580a98f0 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.486+0000 7f603f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.486+0000 7f603f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.486+0000 7f603f7fe640 10 monclient: dump: 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:47.488 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:47.489 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:47.489 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:47.489 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:47.489 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.486+0000 7f60557fa640 4 set_mon_vals no callback set 2024-03-25T18:25:47.489 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.487+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f6048003c40 con 0x7f60580a98f0 2024-03-25T18:25:47.489 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.488+0000 7f60567fc640 10 monclient: get_auth_request con 0x7f604406de00 auth_method 0 2024-03-25T18:25:47.490 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.488+0000 7f603f7fe640 10 client.4656.objecter ms_handle_connect 0x7f604406de00 2024-03-25T18:25:47.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.493+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f6048054950 con 0x7f60580a98f0 2024-03-25T18:25:47.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.493+0000 7f603f7fe640 10 client.4656.objecter ms_dispatch 0x7f6058000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:47.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.493+0000 7f603f7fe640 3 client.4656.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:47.494 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.493+0000 7f603f7fe640 3 client.4656.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:47.495 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.493+0000 7f603f7fe640 20 client.4656.objecter dump_active .. 0 homeless 2024-03-25T18:25:47.495 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.493+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6048050460 con 0x7f60580a98f0 2024-03-25T18:25:47.495 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.493+0000 7f603f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:47.495 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.493+0000 7f603f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:47.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.747+0000 7f605d4d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:47.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.747+0000 7f605d4d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:47.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.747+0000 7f605d4d3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:25:47.749 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.747+0000 7f605d4d3640 1 -- 172.21.15.122:0/3157486012 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f605806f3d0 con 0x7f60580a98f0 2024-03-25T18:25:47.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.788+0000 7f603f7fe640 1 -- 172.21.15.122:0/3157486012 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f6048052390 con 0x7f60580a98f0 2024-03-25T18:25:47.789 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.788+0000 7f603f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:47.790 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.788+0000 7f603f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:47.790 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:25:47.790 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:47.792 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.790+0000 7f605d4d3640 10 librados: watch_flush enter 2024-03-25T18:25:47.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.790+0000 7f605d4d3640 10 librados: watch_flush exit 2024-03-25T18:25:47.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.790+0000 7f605d4d3640 20 client.4656.objecter shutdown clearing up homeless session... 2024-03-25T18:25:47.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.790+0000 7f605d4d3640 10 client.4656.objecter successfully canceled tick 2024-03-25T18:25:47.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.790+0000 7f605d4d3640 1 -- 172.21.15.122:0/3157486012 >> v1:172.21.15.204:6800/676228005 conn(0x7f604406de00 legacy=0x7f60440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.790+0000 7f605d4d3640 10 monclient: shutdown 2024-03-25T18:25:47.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.790+0000 7f605d4d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:47.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.790+0000 7f605d4d3640 1 -- 172.21.15.122:0/3157486012 >> v1:172.21.15.122:6789/0 conn(0x7f60580a98f0 legacy=0x7f6058070be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:47.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.790+0000 7f605d4d3640 1 -- 172.21.15.122:0/3157486012 shutdown_connections 2024-03-25T18:25:47.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.790+0000 7f605d4d3640 1 -- 172.21.15.122:0/3157486012 >> 172.21.15.122:0/3157486012 conn(0x7f60580b7ea0 msgr2=0x7f6058074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:47.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.791+0000 7f605d4d3640 1 -- 172.21.15.122:0/3157486012 shutdown_connections 2024-03-25T18:25:47.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.791+0000 7f605d4d3640 1 -- 172.21.15.122:0/3157486012 wait complete. 2024-03-25T18:25:47.793 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:47.791+0000 7f605d4d3640 1 librados: shutdown 2024-03-25T18:25:47.803 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:52.673 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:09:46.416 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:52.682 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:09:46.425 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:52.713 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:09:46.455 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:52.744 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:09:46.486 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:52.752 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:09:46.498 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:52.756 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:09:46.498 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:52.773 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:09:46.517 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:52.793 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:09:46.535 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:52.795 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:09:46.541 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:54.610 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:54.727 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.726+0000 7f2102fe8640 10 monclient: get_monmap_and_config 2024-03-25T18:25:54.727 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.726+0000 7f2102fe8640 10 monclient: build_initial_monmap 2024-03-25T18:25:54.728 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.726+0000 7f2102fe8640 10 monclient: monmap: 2024-03-25T18:25:54.728 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:25:54.728 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:54.728 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:25:54.727573+0000 2024-03-25T18:25:54.728 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:25:54.727573+0000 2024-03-25T18:25:54.728 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:54.728 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:54.728 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:25:54.728 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:25:54.728 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:25:54.728 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:54.729 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.727+0000 7f2102fe8640 1 Processor -- start 2024-03-25T18:25:54.729 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.727+0000 7f2102fe8640 1 -- start start 2024-03-25T18:25:54.729 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.727+0000 7f2102fe8640 10 monclient: init 2024-03-25T18:25:54.730 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.728+0000 7f2102fe8640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:54.730 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.728+0000 7f2102fe8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:54.730 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.728+0000 7f2102fe8640 10 monclient(hunting): picked mon.noname-c con 0x7f20fc0b4d60 addr v1:172.21.15.204:6789/0 2024-03-25T18:25:54.730 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.728+0000 7f2102fe8640 10 monclient(hunting): picked mon.noname-a con 0x7f20fc0b1730 addr v1:172.21.15.122:6789/0 2024-03-25T18:25:54.730 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.728+0000 7f2102fe8640 10 monclient(hunting): picked mon.noname-b con 0x7f20fc14be90 addr v1:172.21.15.122:6790/0 2024-03-25T18:25:54.730 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.728+0000 7f2102fe8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20fc0af940 con 0x7f20fc0b1730 2024-03-25T18:25:54.730 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.728+0000 7f2102fe8640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20fc0b0a70 con 0x7f20fc0b4d60 2024-03-25T18:25:54.730 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.728+0000 7f2102fe8640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20fc0b0d50 con 0x7f20fc14be90 2024-03-25T18:25:54.730 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.728+0000 7f2102fe8640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:54.730 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.728+0000 7f2102fe8640 10 monclient(hunting): authenticate will time out at 7877.994747s 2024-03-25T18:25:54.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.738+0000 7f2101585640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f20fc14be90 0x7f20fc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:47538/0 (socket says 172.21.15.122:47538) 2024-03-25T18:25:54.739 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.738+0000 7f2101585640 1 -- 172.21.15.122:0/2903327812 learned_addr learned my addr 172.21.15.122:0/2903327812 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:25:54.747 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.746+0000 7f20fb7fe640 1 -- 172.21.15.122:0/2903327812 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1612172287 0 0) 0x7f20fc0b0a70 con 0x7f20fc0b4d60 2024-03-25T18:25:54.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.746+0000 7f20fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:54.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.746+0000 7f20fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:54.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.746+0000 7f20fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:54.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.746+0000 7f20fb7fe640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:25:54.748 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.746+0000 7f20fb7fe640 1 -- 172.21.15.122:0/2903327812 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20ec003520 con 0x7f20fc0b4d60 2024-03-25T18:25:54.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.764+0000 7f20fb7fe640 1 -- 172.21.15.122:0/2903327812 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3286809168 0 0) 0x7f20fc0af940 con 0x7f20fc0b1730 2024-03-25T18:25:54.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.764+0000 7f20fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:54.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.764+0000 7f20fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:54.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.764+0000 7f20fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:54.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.764+0000 7f20fb7fe640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:25:54.765 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.764+0000 7f20fb7fe640 1 -- 172.21.15.122:0/2903327812 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20fc0b0a70 con 0x7f20fc0b1730 2024-03-25T18:25:54.768 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.767+0000 7f20fb7fe640 1 -- 172.21.15.122:0/2903327812 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 750100744 0 0) 0x7f20fc0b0d50 con 0x7f20fc14be90 2024-03-25T18:25:54.768 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.767+0000 7f20fb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:54.769 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.767+0000 7f20fb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:54.769 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.767+0000 7f20fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:54.769 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.767+0000 7f20fb7fe640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:25:54.769 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.767+0000 7f20fb7fe640 1 -- 172.21.15.122:0/2903327812 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20fc0af940 con 0x7f20fc14be90 2024-03-25T18:25:54.770 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.768+0000 7f20fb7fe640 1 -- 172.21.15.122:0/2903327812 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1192016972 0 0) 0x7f20ec003520 con 0x7f20fc0b4d60 2024-03-25T18:25:54.771 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.769+0000 7f20fb7fe640 1 -- 172.21.15.122:0/2903327812 >> v1:172.21.15.122:6790/0 conn(0x7f20fc14be90 legacy=0x7f20fc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:54.771 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.769+0000 7f20fb7fe640 1 -- 172.21.15.122:0/2903327812 >> v1:172.21.15.122:6789/0 conn(0x7f20fc0b1730 legacy=0x7f20fc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:54.771 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.769+0000 7f20fb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:54.771 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.769+0000 7f20fb7fe640 1 monclient: found mon.noname-c 2024-03-25T18:25:54.771 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.769+0000 7f20fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:54.771 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.769+0000 7f20fb7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:25:54.771 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.769+0000 7f20fb7fe640 1 -- 172.21.15.122:0/2903327812 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20fc0b1030 con 0x7f20fc0b4d60 2024-03-25T18:25:54.771 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.769+0000 7f20fb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:54.771 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.769+0000 7f20fb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:54.771 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.769+0000 7f20fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:54.771 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.769+0000 7f20fb7fe640 1 -- 172.21.15.122:0/2903327812 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f20f0001690 con 0x7f20fc0b4d60 2024-03-25T18:25:54.771 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.769+0000 7f20fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:54.771 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.769+0000 7f20fb7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:54.771 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.769+0000 7f20fb7fe640 10 monclient: dump: 2024-03-25T18:25:54.772 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:54.772 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:54.772 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:54.772 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:54.772 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:54.772 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:54.772 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:54.772 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:54.772 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:54.772 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:54.772 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.769+0000 7f2102fe8640 5 monclient: authenticate success, global_id 4648 2024-03-25T18:25:54.772 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.769+0000 7f2102fe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:54.782 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.781+0000 7f20fb7fe640 1 -- 172.21.15.122:0/2903327812 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20f00017f0 con 0x7f20fc0b4d60 2024-03-25T18:25:54.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.781+0000 7f20fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:54.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.781+0000 7f20fb7fe640 1 -- 172.21.15.122:0/2903327812 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f20f0001c00 con 0x7f20fc0b4d60 2024-03-25T18:25:54.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.781+0000 7f2102fe8640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:54.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.781+0000 7f2102fe8640 4 set_mon_vals no callback set 2024-03-25T18:25:54.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.781+0000 7f20fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:54.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.781+0000 7f20fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:54.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.781+0000 7f20fb7fe640 10 monclient: dump: 2024-03-25T18:25:54.783 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:54.783 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:54.783 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:54.783 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:54.783 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:54.783 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:54.783 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:54.784 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:54.784 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:54.784 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:54.786 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.782+0000 7f2102fe8640 10 monclient: shutdown 2024-03-25T18:25:54.786 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.782+0000 7f2102fe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:54.786 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.782+0000 7f2102fe8640 1 -- 172.21.15.122:0/2903327812 >> v1:172.21.15.204:6789/0 conn(0x7f20fc0b4d60 legacy=0x7f20fc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:54.786 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.782+0000 7f2102fe8640 1 -- 172.21.15.122:0/2903327812 shutdown_connections 2024-03-25T18:25:54.786 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.782+0000 7f2102fe8640 1 -- 172.21.15.122:0/2903327812 >> 172.21.15.122:0/2903327812 conn(0x7f20fc0617a0 msgr2=0x7f20fc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:54.786 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.782+0000 7f2102fe8640 1 -- 172.21.15.122:0/2903327812 shutdown_connections 2024-03-25T18:25:54.786 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.782+0000 7f2102fe8640 1 -- 172.21.15.122:0/2903327812 wait complete. 2024-03-25T18:25:54.786 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.783+0000 7f2102fe8640 10 monclient: build_initial_monmap 2024-03-25T18:25:54.786 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.783+0000 7f2102fe8640 10 monclient: monmap: 2024-03-25T18:25:54.786 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:25:54.786 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:54.786 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:25:54.787 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:25:54.787 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:54.787 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:25:54.787 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:25:54.787 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:25:54.787 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:25:54.787 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:54.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.783+0000 7f2102fe8640 1 librados: starting msgr at 2024-03-25T18:25:54.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.783+0000 7f2102fe8640 1 librados: starting objecter 2024-03-25T18:25:54.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.783+0000 7f2102fe8640 1 Processor -- start 2024-03-25T18:25:54.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.783+0000 7f2102fe8640 1 -- start start 2024-03-25T18:25:54.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.783+0000 7f2102fe8640 1 librados: setting wanted keys 2024-03-25T18:25:54.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.783+0000 7f2102fe8640 1 librados: calling monclient init 2024-03-25T18:25:54.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.783+0000 7f2102fe8640 10 monclient: init 2024-03-25T18:25:54.787 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.784+0000 7f2102fe8640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:54.788 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.784+0000 7f2102fe8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:54.788 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.784+0000 7f2102fe8640 10 monclient(hunting): picked mon.noname-a con 0x7f20fc0b1730 addr v1:172.21.15.122:6789/0 2024-03-25T18:25:54.788 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.784+0000 7f2102fe8640 10 monclient(hunting): picked mon.noname-c con 0x7f20fc0b4d60 addr v1:172.21.15.122:6790/0 2024-03-25T18:25:54.788 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.784+0000 7f2102fe8640 10 monclient(hunting): picked mon.noname-b con 0x7f20fc14be90 addr v1:172.21.15.204:6789/0 2024-03-25T18:25:54.788 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.784+0000 7f2102fe8640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20fc1d7940 con 0x7f20fc0b1730 2024-03-25T18:25:54.788 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.784+0000 7f2102fe8640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20fc1382f0 con 0x7f20fc14be90 2024-03-25T18:25:54.788 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.784+0000 7f2102fe8640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20fc137320 con 0x7f20fc0b4d60 2024-03-25T18:25:54.788 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.784+0000 7f2102fe8640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:54.788 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.784+0000 7f2102fe8640 10 monclient(hunting): authenticate will time out at 7878.050790s 2024-03-25T18:25:54.795 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.794+0000 7f2100d84640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f20fc0b1730 0x7f20fc136c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51370/0 (socket says 172.21.15.122:51370) 2024-03-25T18:25:54.796 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.794+0000 7f2100d84640 1 -- 172.21.15.122:0/1364990719 learned_addr learned my addr 172.21.15.122:0/1364990719 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:25:54.802 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.801+0000 7f20f97fa640 10 client.?.objecter ms_handle_connect 0x7f20fc0b4d60 2024-03-25T18:25:54.802 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.801+0000 7f20f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:54.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.803+0000 7f20f97fa640 10 client.?.objecter ms_handle_connect 0x7f20fc0b1730 2024-03-25T18:25:54.804 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.803+0000 7f20f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:54.805 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.804+0000 7f20f97fa640 10 client.?.objecter ms_handle_connect 0x7f20fc14be90 2024-03-25T18:25:54.805 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.804+0000 7f20f97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:54.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.807+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4031079088 0 0) 0x7f20fc137320 con 0x7f20fc0b4d60 2024-03-25T18:25:54.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.807+0000 7f20f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:54.808 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.807+0000 7f20f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:54.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.807+0000 7f20f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:54.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.807+0000 7f20f97fa640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:25:54.809 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.807+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e0004eb0 con 0x7f20fc0b4d60 2024-03-25T18:25:54.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.814+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1845606084 0 0) 0x7f20e0004eb0 con 0x7f20fc0b4d60 2024-03-25T18:25:54.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.814+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20fc137320 con 0x7f20fc0b4d60 2024-03-25T18:25:54.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.814+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f20e8001970 con 0x7f20fc0b4d60 2024-03-25T18:25:54.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.814+0000 7f20f97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:54.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.814+0000 7f20f97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:54.816 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.814+0000 7f20f97fa640 10 monclient(hunting): dump: 2024-03-25T18:25:54.816 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:54.816 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:54.816 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:54.816 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:54.816 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:54.816 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:54.816 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:54.817 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:54.817 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:54.817 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:54.817 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.814+0000 7f20f97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:54.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.817+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 680464810 0 0) 0x7f20fc1d7940 con 0x7f20fc0b1730 2024-03-25T18:25:54.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.817+0000 7f20f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:54.818 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.817+0000 7f20f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:54.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.817+0000 7f20f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:54.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.817+0000 7f20f97fa640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:25:54.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.817+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e0004eb0 con 0x7f20fc0b1730 2024-03-25T18:25:54.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.818+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1141125348 0 0) 0x7f20fc1382f0 con 0x7f20fc14be90 2024-03-25T18:25:54.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.818+0000 7f20f97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:54.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.818+0000 7f20f97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:54.819 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.818+0000 7f20f97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:54.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.818+0000 7f20f97fa640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:25:54.820 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.818+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20fc1d7940 con 0x7f20fc14be90 2024-03-25T18:25:54.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2094327547 0 0) 0x7f20fc137320 con 0x7f20fc0b4d60 2024-03-25T18:25:54.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 >> v1:172.21.15.204:6789/0 conn(0x7f20fc14be90 legacy=0x7f20fc12fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:54.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 >> v1:172.21.15.122:6789/0 conn(0x7f20fc0b1730 legacy=0x7f20fc136c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:54.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f20f97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:25:54.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f20f97fa640 1 monclient: found mon.c 2024-03-25T18:25:54.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f20f97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:54.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f20f97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:54.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20fc1d8530 con 0x7f20fc0b4d60 2024-03-25T18:25:54.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f20f97fa640 10 monclient: _finish_auth 0 2024-03-25T18:25:54.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f20f97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:25:54.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f20f97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:54.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f2102fe8640 5 monclient: authenticate success, global_id 4619 2024-03-25T18:25:54.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f2102fe8640 10 monclient: _renew_subs 2024-03-25T18:25:54.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f2102fe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:54.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f2102fe8640 1 -- 172.21.15.122:0/1364990719 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20fc130570 con 0x7f20fc0b4d60 2024-03-25T18:25:54.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f2102fe8640 10 client.4619.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:54.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f2102fe8640 10 monclient: _renew_subs 2024-03-25T18:25:54.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f2102fe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:54.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f2102fe8640 1 -- 172.21.15.122:0/1364990719 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20fc143ae0 con 0x7f20fc0b4d60 2024-03-25T18:25:54.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.825+0000 7f2102fe8640 1 librados: init done 2024-03-25T18:25:54.829 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.826+0000 7f2102fe8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:54.829 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.826+0000 7f2102fe8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:54.829 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.826+0000 7f2102fe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:54.829 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:54.826+0000 7f2102fe8640 1 -- 172.21.15.122:0/1364990719 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20fc1d77a0 con 0x7f20fc0b4d60 2024-03-25T18:25:55.042 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.041+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20e8003610 con 0x7f20fc0b4d60 2024-03-25T18:25:55.042 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.041+0000 7f20f97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:55.042 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.041+0000 7f20faffd640 4 set_mon_vals no callback set 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.118+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f20e80037c0 con 0x7f20fc0b4d60 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.118+0000 7f20f97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.118+0000 7f20f97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.118+0000 7f20f97fa640 10 monclient: dump: 2024-03-25T18:25:55.119 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:25:55.120 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:25:55.120 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:55.120 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:25:55.120 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:55.120 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:25:55.120 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:25:55.120 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:25:55.120 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:25:55.120 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:25:55.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.119+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f20e8019810 con 0x7f20fc0b4d60 2024-03-25T18:25:55.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.119+0000 7f2100d84640 10 monclient: get_auth_request con 0x7f20e006de80 auth_method 0 2024-03-25T18:25:55.121 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.120+0000 7f20f97fa640 10 client.4619.objecter ms_handle_connect 0x7f20e006de80 2024-03-25T18:25:55.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.126+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f20e80534d0 con 0x7f20fc0b4d60 2024-03-25T18:25:55.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.126+0000 7f20f97fa640 10 client.4619.objecter ms_dispatch 0x7f20fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:25:55.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.126+0000 7f20f97fa640 3 client.4619.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:25:55.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.126+0000 7f20f97fa640 3 client.4619.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:25:55.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.126+0000 7f20f97fa640 20 client.4619.objecter dump_active .. 0 homeless 2024-03-25T18:25:55.127 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.126+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20e804ee10 con 0x7f20fc0b4d60 2024-03-25T18:25:55.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.126+0000 7f20f97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:55.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.126+0000 7f20f97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:55.369 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.368+0000 7f2102fe8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:55.369 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.368+0000 7f2102fe8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:55.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.368+0000 7f2102fe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:25:55.370 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.368+0000 7f2102fe8640 1 -- 172.21.15.122:0/1364990719 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f20fc1d86c0 con 0x7f20fc0b4d60 2024-03-25T18:25:55.455 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.454+0000 7f20f97fa640 1 -- 172.21.15.122:0/1364990719 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f20fc1d86c0 con 0x7f20fc0b4d60 2024-03-25T18:25:55.455 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.454+0000 7f20f97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:55.456 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.454+0000 7f20f97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:55.456 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:25:55.456 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:55.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.456+0000 7f2102fe8640 10 librados: watch_flush enter 2024-03-25T18:25:55.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.456+0000 7f2102fe8640 10 librados: watch_flush exit 2024-03-25T18:25:55.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.456+0000 7f2102fe8640 20 client.4619.objecter shutdown clearing up homeless session... 2024-03-25T18:25:55.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.456+0000 7f2102fe8640 10 client.4619.objecter successfully canceled tick 2024-03-25T18:25:55.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.456+0000 7f2102fe8640 1 -- 172.21.15.122:0/1364990719 >> v1:172.21.15.204:6800/676228005 conn(0x7f20e006de80 legacy=0x7f20e0070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:55.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.456+0000 7f2102fe8640 10 monclient: shutdown 2024-03-25T18:25:55.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.456+0000 7f2102fe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:55.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.456+0000 7f2102fe8640 1 -- 172.21.15.122:0/1364990719 >> v1:172.21.15.122:6790/0 conn(0x7f20fc0b4d60 legacy=0x7f20fc1d7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:55.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.456+0000 7f2102fe8640 1 -- 172.21.15.122:0/1364990719 shutdown_connections 2024-03-25T18:25:55.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.456+0000 7f2102fe8640 1 -- 172.21.15.122:0/1364990719 >> 172.21.15.122:0/1364990719 conn(0x7f20fc0617a0 msgr2=0x7f20fc0b2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:55.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.456+0000 7f2102fe8640 1 -- 172.21.15.122:0/1364990719 shutdown_connections 2024-03-25T18:25:55.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.456+0000 7f2102fe8640 1 -- 172.21.15.122:0/1364990719 wait complete. 2024-03-25T18:25:55.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:25:55.456+0000 7f2102fe8640 1 librados: shutdown 2024-03-25T18:25:55.469 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:02.276 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:02.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.393+0000 7f2866863640 10 monclient: get_monmap_and_config 2024-03-25T18:26:02.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.393+0000 7f2866863640 10 monclient: build_initial_monmap 2024-03-25T18:26:02.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.393+0000 7f2866863640 10 monclient: monmap: 2024-03-25T18:26:02.395 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:26:02.395 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:02.395 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:26:02.394219+0000 2024-03-25T18:26:02.395 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:26:02.394219+0000 2024-03-25T18:26:02.395 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:02.395 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:02.395 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:26:02.395 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:26:02.395 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:26:02.395 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:02.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.394+0000 7f2866863640 1 Processor -- start 2024-03-25T18:26:02.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.394+0000 7f2866863640 1 -- start start 2024-03-25T18:26:02.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.394+0000 7f2866863640 10 monclient: init 2024-03-25T18:26:02.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.395+0000 7f2866863640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:02.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.395+0000 7f2866863640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:02.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.395+0000 7f2866863640 10 monclient(hunting): picked mon.noname-b con 0x7f28600a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:26:02.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.395+0000 7f2866863640 10 monclient(hunting): picked mon.noname-c con 0x7f286013f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:26:02.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.395+0000 7f2866863640 10 monclient(hunting): picked mon.noname-a con 0x7f28601430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:26:02.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.395+0000 7f2866863640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2860142520 con 0x7f28601430c0 2024-03-25T18:26:02.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.395+0000 7f2866863640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28600b28a0 con 0x7f286013f710 2024-03-25T18:26:02.397 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.395+0000 7f2866863640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28600b2a30 con 0x7f28600a98f0 2024-03-25T18:26:02.398 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.395+0000 7f2866863640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:02.398 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.395+0000 7f2866863640 10 monclient(hunting): authenticate will time out at 7885.661520s 2024-03-25T18:26:02.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.406+0000 7f2864e00640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f28601430c0 0x7f286014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:56564/0 (socket says 172.21.15.122:56564) 2024-03-25T18:26:02.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.406+0000 7f2864e00640 1 -- 172.21.15.122:0/2922441516 learned_addr learned my addr 172.21.15.122:0/2922441516 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:26:02.421 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.420+0000 7f285effd640 1 -- 172.21.15.122:0/2922441516 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994684584 0 0) 0x7f28600b28a0 con 0x7f286013f710 2024-03-25T18:26:02.421 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.420+0000 7f285effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.421 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.420+0000 7f285effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.421 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.420+0000 7f285effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.421 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.420+0000 7f285effd640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:26:02.421 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.420+0000 7f285effd640 1 -- 172.21.15.122:0/2922441516 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2848003520 con 0x7f286013f710 2024-03-25T18:26:02.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.432+0000 7f285effd640 1 -- 172.21.15.122:0/2922441516 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204694278 0 0) 0x7f28600b2a30 con 0x7f28600a98f0 2024-03-25T18:26:02.433 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.432+0000 7f285effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.432+0000 7f285effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.432+0000 7f285effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.432+0000 7f285effd640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:26:02.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.432+0000 7f285effd640 1 -- 172.21.15.122:0/2922441516 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28600b28a0 con 0x7f28600a98f0 2024-03-25T18:26:02.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.434+0000 7f285effd640 1 -- 172.21.15.122:0/2922441516 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060499935 0 0) 0x7f2860142520 con 0x7f28601430c0 2024-03-25T18:26:02.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.434+0000 7f285effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.434+0000 7f285effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.434+0000 7f285effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.434+0000 7f285effd640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:26:02.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.434+0000 7f285effd640 1 -- 172.21.15.122:0/2922441516 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28600b2a30 con 0x7f28601430c0 2024-03-25T18:26:02.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.447+0000 7f285effd640 1 -- 172.21.15.122:0/2922441516 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3754617070 0 0) 0x7f2848003520 con 0x7f286013f710 2024-03-25T18:26:02.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.447+0000 7f285effd640 1 -- 172.21.15.122:0/2922441516 >> v1:172.21.15.122:6790/0 conn(0x7f28600a98f0 legacy=0x7f28600a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.447+0000 7f285effd640 1 -- 172.21.15.122:0/2922441516 >> v1:172.21.15.122:6789/0 conn(0x7f28601430c0 legacy=0x7f286014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.447+0000 7f285effd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:02.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.447+0000 7f285effd640 1 monclient: found mon.noname-c 2024-03-25T18:26:02.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.447+0000 7f285effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:02.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.447+0000 7f285effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:26:02.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.447+0000 7f285effd640 1 -- 172.21.15.122:0/2922441516 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28600b44c0 con 0x7f286013f710 2024-03-25T18:26:02.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.447+0000 7f285effd640 10 monclient: _finish_auth 0 2024-03-25T18:26:02.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.447+0000 7f285effd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:02.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.447+0000 7f285effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:02.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.447+0000 7f285effd640 1 -- 172.21.15.122:0/2922441516 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f2854001690 con 0x7f286013f710 2024-03-25T18:26:02.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.447+0000 7f285effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.447+0000 7f285effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:02.450 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.447+0000 7f285effd640 10 monclient: dump: 2024-03-25T18:26:02.450 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:02.450 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:02.450 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:02.450 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:02.450 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.450 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:02.450 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:02.450 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:02.451 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:02.451 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:02.451 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.447+0000 7f2866863640 5 monclient: authenticate success, global_id 4654 2024-03-25T18:26:02.451 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.447+0000 7f2866863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:02.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.463+0000 7f285effd640 1 -- 172.21.15.122:0/2922441516 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28540017f0 con 0x7f286013f710 2024-03-25T18:26:02.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.463+0000 7f285effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:02.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.463+0000 7f285effd640 1 -- 172.21.15.122:0/2922441516 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f2854001c00 con 0x7f286013f710 2024-03-25T18:26:02.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.463+0000 7f285effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.463+0000 7f285effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:02.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.463+0000 7f285effd640 10 monclient: dump: 2024-03-25T18:26:02.465 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:02.465 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:02.465 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:02.465 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:02.465 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.466 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:02.466 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:02.466 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:02.466 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:02.466 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:02.466 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.463+0000 7f2866863640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:02.466 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.463+0000 7f2866863640 4 set_mon_vals no callback set 2024-03-25T18:26:02.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.464+0000 7f2866863640 10 monclient: shutdown 2024-03-25T18:26:02.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.464+0000 7f2866863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:02.468 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.464+0000 7f2866863640 1 -- 172.21.15.122:0/2922441516 >> v1:172.21.15.204:6789/0 conn(0x7f286013f710 legacy=0x7f28601429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.464+0000 7f2866863640 1 -- 172.21.15.122:0/2922441516 shutdown_connections 2024-03-25T18:26:02.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.464+0000 7f2866863640 1 -- 172.21.15.122:0/2922441516 >> 172.21.15.122:0/2922441516 conn(0x7f28600b7ea0 msgr2=0x7f28600b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:02.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.464+0000 7f2866863640 1 -- 172.21.15.122:0/2922441516 shutdown_connections 2024-03-25T18:26:02.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.465+0000 7f2866863640 1 -- 172.21.15.122:0/2922441516 wait complete. 2024-03-25T18:26:02.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.465+0000 7f2866863640 10 monclient: build_initial_monmap 2024-03-25T18:26:02.469 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.465+0000 7f2866863640 10 monclient: monmap: 2024-03-25T18:26:02.469 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:26:02.469 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:02.469 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:26:02.469 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:26:02.469 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:02.469 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:26:02.469 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:26:02.469 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:26:02.469 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:26:02.470 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:02.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.466+0000 7f2866863640 1 librados: starting msgr at 2024-03-25T18:26:02.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.466+0000 7f2866863640 1 librados: starting objecter 2024-03-25T18:26:02.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.466+0000 7f2866863640 1 Processor -- start 2024-03-25T18:26:02.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.466+0000 7f2866863640 1 -- start start 2024-03-25T18:26:02.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.466+0000 7f2866863640 1 librados: setting wanted keys 2024-03-25T18:26:02.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.466+0000 7f2866863640 1 librados: calling monclient init 2024-03-25T18:26:02.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.466+0000 7f2866863640 10 monclient: init 2024-03-25T18:26:02.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.467+0000 7f2866863640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:02.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.467+0000 7f2866863640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:02.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.467+0000 7f2866863640 10 monclient(hunting): picked mon.noname-c con 0x7f28600a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:26:02.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.467+0000 7f2866863640 10 monclient(hunting): picked mon.noname-a con 0x7f286013f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:26:02.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.467+0000 7f2866863640 10 monclient(hunting): picked mon.noname-b con 0x7f28601430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:26:02.470 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.467+0000 7f2866863640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28601d7490 con 0x7f286013f710 2024-03-25T18:26:02.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.467+0000 7f2866863640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28601d76c0 con 0x7f28601430c0 2024-03-25T18:26:02.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.467+0000 7f2866863640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28601d82f0 con 0x7f28600a98f0 2024-03-25T18:26:02.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.467+0000 7f2866863640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:02.471 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.467+0000 7f2866863640 10 monclient(hunting): authenticate will time out at 7885.733284s 2024-03-25T18:26:02.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.482+0000 7f2864e00640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f28601430c0 0x7f2860133fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:49300/0 (socket says 172.21.15.122:49300) 2024-03-25T18:26:02.483 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.482+0000 7f2864e00640 1 -- 172.21.15.122:0/56919892 learned_addr learned my addr 172.21.15.122:0/56919892 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:26:02.484 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.483+0000 7f285cff9640 10 client.?.objecter ms_handle_connect 0x7f28601430c0 2024-03-25T18:26:02.484 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.483+0000 7f285cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:02.487 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.486+0000 7f285cff9640 10 client.?.objecter ms_handle_connect 0x7f28600a98f0 2024-03-25T18:26:02.487 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.486+0000 7f285cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:02.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.487+0000 7f285cff9640 10 client.?.objecter ms_handle_connect 0x7f286013f710 2024-03-25T18:26:02.488 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.487+0000 7f285cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:02.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.494+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1024260557 0 0) 0x7f28601d76c0 con 0x7f28601430c0 2024-03-25T18:26:02.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.494+0000 7f285cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.494+0000 7f285cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.494+0000 7f285cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.494+0000 7f285cff9640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:26:02.496 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.494+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f284c004eb0 con 0x7f28601430c0 2024-03-25T18:26:02.499 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.497+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3284211401 0 0) 0x7f28601d82f0 con 0x7f28600a98f0 2024-03-25T18:26:02.499 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.497+0000 7f285cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.499 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.497+0000 7f285cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.499 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.497+0000 7f285cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.499 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.497+0000 7f285cff9640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:26:02.499 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.497+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28601d76c0 con 0x7f28600a98f0 2024-03-25T18:26:02.500 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.499+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445592665 0 0) 0x7f28601d7490 con 0x7f286013f710 2024-03-25T18:26:02.500 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.499+0000 7f285cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:02.500 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.499+0000 7f285cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:02.501 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.499+0000 7f285cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:02.501 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.499+0000 7f285cff9640 10 monclient(hunting): my global_id is 4668 2024-03-25T18:26:02.501 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.499+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28601d82f0 con 0x7f286013f710 2024-03-25T18:26:02.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.504+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2117632958 0 0) 0x7f284c004eb0 con 0x7f28601430c0 2024-03-25T18:26:02.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.504+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28601d7490 con 0x7f28601430c0 2024-03-25T18:26:02.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.504+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f2858001bd0 con 0x7f28601430c0 2024-03-25T18:26:02.505 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.504+0000 7f285cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.504+0000 7f285cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:02.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.504+0000 7f285cff9640 10 monclient(hunting): dump: 2024-03-25T18:26:02.506 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:02.506 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:02.506 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:02.506 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:02.506 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.506 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:02.506 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:02.506 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:02.506 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:02.506 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:02.506 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.504+0000 7f285cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:02.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.524+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3228497671 0 0) 0x7f28601d76c0 con 0x7f28600a98f0 2024-03-25T18:26:02.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.524+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f284c004eb0 con 0x7f28600a98f0 2024-03-25T18:26:02.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.524+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f2850001970 con 0x7f28600a98f0 2024-03-25T18:26:02.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.524+0000 7f285cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.524+0000 7f285cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:02.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.524+0000 7f285cff9640 10 monclient(hunting): dump: 2024-03-25T18:26:02.526 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:02.526 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:02.526 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:02.526 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:02.526 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.526 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:02.526 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:02.527 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:02.527 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:02.527 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:02.531 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.529+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2849581460 0 0) 0x7f284c004eb0 con 0x7f28600a98f0 2024-03-25T18:26:02.531 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 >> v1:172.21.15.204:6789/0 conn(0x7f28601430c0 legacy=0x7f2860133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.531 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 >> v1:172.21.15.122:6789/0 conn(0x7f286013f710 legacy=0x7f28601383e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:02.531 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f285cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:02.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f285cff9640 1 monclient: found mon.c 2024-03-25T18:26:02.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f285cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:02.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f285cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:02.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2860138af0 con 0x7f28600a98f0 2024-03-25T18:26:02.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f285cff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:02.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f285cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:02.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f285cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:02.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f2866863640 5 monclient: authenticate success, global_id 4625 2024-03-25T18:26:02.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f2866863640 10 monclient: _renew_subs 2024-03-25T18:26:02.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f2866863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:02.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f2866863640 1 -- 172.21.15.122:0/56919892 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28601376d0 con 0x7f28600a98f0 2024-03-25T18:26:02.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f2866863640 10 client.4625.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:02.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f2866863640 10 monclient: _renew_subs 2024-03-25T18:26:02.532 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f2866863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:02.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f2866863640 1 -- 172.21.15.122:0/56919892 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2860134fc0 con 0x7f28600a98f0 2024-03-25T18:26:02.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.530+0000 7f2866863640 1 librados: init done 2024-03-25T18:26:02.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.531+0000 7f2866863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:02.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.531+0000 7f2866863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:02.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.531+0000 7f2866863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:02.533 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.531+0000 7f2866863640 1 -- 172.21.15.122:0/56919892 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28601d7850 con 0x7f28600a98f0 2024-03-25T18:26:02.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.865+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2850003610 con 0x7f28600a98f0 2024-03-25T18:26:02.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.865+0000 7f285cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:02.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.865+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f2850003920 con 0x7f28600a98f0 2024-03-25T18:26:02.867 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.865+0000 7f285cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:02.867 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.865+0000 7f285cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:02.867 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.865+0000 7f285cff9640 10 monclient: dump: 2024-03-25T18:26:02.867 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:02.867 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:02.867 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:02.867 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:02.867 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:02.867 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:02.867 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:02.867 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:02.867 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:02.867 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:02.867 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.865+0000 7f285e7fc640 4 set_mon_vals no callback set 2024-03-25T18:26:02.868 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.865+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f28500198f0 con 0x7f28600a98f0 2024-03-25T18:26:02.868 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.866+0000 7f285f7fe640 10 monclient: get_auth_request con 0x7f284c06df10 auth_method 0 2024-03-25T18:26:02.868 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.866+0000 7f285cff9640 10 client.4625.objecter ms_handle_connect 0x7f284c06df10 2024-03-25T18:26:02.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.871+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f28500535b0 con 0x7f28600a98f0 2024-03-25T18:26:02.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.871+0000 7f285cff9640 10 client.4625.objecter ms_dispatch 0x7f2860000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:02.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.871+0000 7f285cff9640 3 client.4625.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:02.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.871+0000 7f285cff9640 3 client.4625.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:02.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.871+0000 7f285cff9640 20 client.4625.objecter dump_active .. 0 homeless 2024-03-25T18:26:02.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.871+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f285004f0c0 con 0x7f28600a98f0 2024-03-25T18:26:02.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.871+0000 7f285cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:02.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:02.871+0000 7f285cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:03.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.130+0000 7f2866863640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:03.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.130+0000 7f2866863640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:03.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.130+0000 7f2866863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:03.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.130+0000 7f2866863640 1 -- 172.21.15.122:0/56919892 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f28600627b0 con 0x7f28600a98f0 2024-03-25T18:26:03.241 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.240+0000 7f285cff9640 1 -- 172.21.15.122:0/56919892 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f2850050ff0 con 0x7f28600a98f0 2024-03-25T18:26:03.242 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.240+0000 7f285cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:03.242 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.240+0000 7f285cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:03.242 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:26:03.242 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:03.244 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.243+0000 7f2866863640 10 librados: watch_flush enter 2024-03-25T18:26:03.245 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.243+0000 7f2866863640 10 librados: watch_flush exit 2024-03-25T18:26:03.245 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.243+0000 7f2866863640 20 client.4625.objecter shutdown clearing up homeless session... 2024-03-25T18:26:03.245 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.243+0000 7f2866863640 10 client.4625.objecter successfully canceled tick 2024-03-25T18:26:03.245 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.243+0000 7f2866863640 1 -- 172.21.15.122:0/56919892 >> v1:172.21.15.204:6800/676228005 conn(0x7f284c06df10 legacy=0x7f284c0703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.245 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.243+0000 7f2866863640 10 monclient: shutdown 2024-03-25T18:26:03.245 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.243+0000 7f2866863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:03.245 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.243+0000 7f2866863640 1 -- 172.21.15.122:0/56919892 >> v1:172.21.15.122:6790/0 conn(0x7f28600a98f0 legacy=0x7f28601d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:03.245 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.243+0000 7f2866863640 1 -- 172.21.15.122:0/56919892 shutdown_connections 2024-03-25T18:26:03.245 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.243+0000 7f2866863640 1 -- 172.21.15.122:0/56919892 >> 172.21.15.122:0/56919892 conn(0x7f28600b7ea0 msgr2=0x7f2860074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:03.245 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.243+0000 7f2866863640 1 -- 172.21.15.122:0/56919892 shutdown_connections 2024-03-25T18:26:03.245 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.243+0000 7f2866863640 1 -- 172.21.15.122:0/56919892 wait complete. 2024-03-25T18:26:03.245 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:03.243+0000 7f2866863640 1 librados: shutdown 2024-03-25T18:26:03.258 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:10.064 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:10.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.180+0000 7fa615dc7640 10 monclient: get_monmap_and_config 2024-03-25T18:26:10.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.180+0000 7fa615dc7640 10 monclient: build_initial_monmap 2024-03-25T18:26:10.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.180+0000 7fa615dc7640 10 monclient: monmap: 2024-03-25T18:26:10.180 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:26:10.180 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:10.180 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:26:10.180159+0000 2024-03-25T18:26:10.180 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:26:10.180159+0000 2024-03-25T18:26:10.180 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:10.181 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:10.181 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:26:10.181 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:26:10.181 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:26:10.181 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:10.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.181+0000 7fa615dc7640 1 Processor -- start 2024-03-25T18:26:10.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.181+0000 7fa615dc7640 1 -- start start 2024-03-25T18:26:10.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.181+0000 7fa615dc7640 10 monclient: init 2024-03-25T18:26:10.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.181+0000 7fa615dc7640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:10.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.181+0000 7fa615dc7640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:10.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.181+0000 7fa615dc7640 10 monclient(hunting): picked mon.noname-a con 0x7fa6100705b0 addr v1:172.21.15.122:6789/0 2024-03-25T18:26:10.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.181+0000 7fa615dc7640 10 monclient(hunting): picked mon.noname-c con 0x7fa6101402d0 addr v1:172.21.15.204:6789/0 2024-03-25T18:26:10.182 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.182+0000 7fa615dc7640 10 monclient(hunting): picked mon.noname-b con 0x7fa610149ae0 addr v1:172.21.15.122:6790/0 2024-03-25T18:26:10.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.182+0000 7fa615dc7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6100718c0 con 0x7fa6100705b0 2024-03-25T18:26:10.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.182+0000 7fa615dc7640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6100714c0 con 0x7fa6101402d0 2024-03-25T18:26:10.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.182+0000 7fa615dc7640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa610072b50 con 0x7fa610149ae0 2024-03-25T18:26:10.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.182+0000 7fa615dc7640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:10.183 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.182+0000 7fa615dc7640 10 monclient(hunting): authenticate will time out at 7893.447136s 2024-03-25T18:26:10.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.190+0000 7fa60f7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fa6100705b0 0x7fa61013d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:55142/0 (socket says 172.21.15.122:55142) 2024-03-25T18:26:10.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.190+0000 7fa60f7fe640 1 -- 172.21.15.122:0/949548964 learned_addr learned my addr 172.21.15.122:0/949548964 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:26:10.213 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.213+0000 7fa60effd640 1 -- 172.21.15.122:0/949548964 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821631647 0 0) 0x7fa6100718c0 con 0x7fa6100705b0 2024-03-25T18:26:10.213 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.213+0000 7fa60effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:10.213 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.213+0000 7fa60effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:10.214 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.213+0000 7fa60effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:10.214 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.213+0000 7fa60effd640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:26:10.214 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.213+0000 7fa60effd640 1 -- 172.21.15.122:0/949548964 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5fc003520 con 0x7fa6100705b0 2024-03-25T18:26:10.218 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.218+0000 7fa60effd640 1 -- 172.21.15.122:0/949548964 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499928061 0 0) 0x7fa6100714c0 con 0x7fa6101402d0 2024-03-25T18:26:10.218 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.218+0000 7fa60effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:10.218 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.218+0000 7fa60effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:10.218 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.218+0000 7fa60effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:10.218 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.218+0000 7fa60effd640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:26:10.219 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.218+0000 7fa60effd640 1 -- 172.21.15.122:0/949548964 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6100718c0 con 0x7fa6101402d0 2024-03-25T18:26:10.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.224+0000 7fa60effd640 1 -- 172.21.15.122:0/949548964 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 196177716 0 0) 0x7fa610072b50 con 0x7fa610149ae0 2024-03-25T18:26:10.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.224+0000 7fa60effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:10.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.224+0000 7fa60effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:10.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.224+0000 7fa60effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:10.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.224+0000 7fa60effd640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:26:10.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.224+0000 7fa60effd640 1 -- 172.21.15.122:0/949548964 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6100714c0 con 0x7fa610149ae0 2024-03-25T18:26:10.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.233+0000 7fa60effd640 1 -- 172.21.15.122:0/949548964 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1032995727 0 0) 0x7fa5fc003520 con 0x7fa6100705b0 2024-03-25T18:26:10.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.233+0000 7fa60effd640 1 -- 172.21.15.122:0/949548964 >> v1:172.21.15.122:6790/0 conn(0x7fa610149ae0 legacy=0x7fa61014bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.233+0000 7fa60effd640 1 -- 172.21.15.122:0/949548964 >> v1:172.21.15.204:6789/0 conn(0x7fa6101402d0 legacy=0x7fa610140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.233+0000 7fa60effd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:10.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.233+0000 7fa60effd640 1 monclient: found mon.noname-a 2024-03-25T18:26:10.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.233+0000 7fa60effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:10.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.233+0000 7fa60effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:26:10.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.233+0000 7fa60effd640 1 -- 172.21.15.122:0/949548964 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61013dd10 con 0x7fa6100705b0 2024-03-25T18:26:10.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.233+0000 7fa60effd640 10 monclient: _finish_auth 0 2024-03-25T18:26:10.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.233+0000 7fa60effd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:10.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.233+0000 7fa60effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:10.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.233+0000 7fa60effd640 1 -- 172.21.15.122:0/949548964 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fa600001690 con 0x7fa6100705b0 2024-03-25T18:26:10.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.233+0000 7fa60effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:10.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.233+0000 7fa60effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:10.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.233+0000 7fa60effd640 10 monclient: dump: 2024-03-25T18:26:10.235 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:10.235 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:10.235 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:10.235 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:10.235 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:10.235 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:10.235 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:10.235 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:10.235 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:10.236 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:10.236 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.233+0000 7fa615dc7640 5 monclient: authenticate success, global_id 4671 2024-03-25T18:26:10.236 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.233+0000 7fa615dc7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:10.268 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.268+0000 7fa60effd640 1 -- 172.21.15.122:0/949548964 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa6000017f0 con 0x7fa6100705b0 2024-03-25T18:26:10.268 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.268+0000 7fa60effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:10.268 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.268+0000 7fa60effd640 1 -- 172.21.15.122:0/949548964 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fa600001c00 con 0x7fa6100705b0 2024-03-25T18:26:10.268 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.268+0000 7fa615dc7640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:10.268 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.268+0000 7fa615dc7640 4 set_mon_vals no callback set 2024-03-25T18:26:10.268 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.268+0000 7fa60effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:10.268 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.268+0000 7fa60effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:10.269 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.268+0000 7fa60effd640 10 monclient: dump: 2024-03-25T18:26:10.269 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:10.269 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:10.269 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:10.269 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:10.269 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:10.269 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:10.269 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:10.269 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:10.269 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:10.269 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:10.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.268+0000 7fa615dc7640 10 monclient: shutdown 2024-03-25T18:26:10.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.268+0000 7fa615dc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:10.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.268+0000 7fa615dc7640 1 -- 172.21.15.122:0/949548964 >> v1:172.21.15.122:6789/0 conn(0x7fa6100705b0 legacy=0x7fa61013d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.268+0000 7fa615dc7640 1 -- 172.21.15.122:0/949548964 shutdown_connections 2024-03-25T18:26:10.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.268+0000 7fa615dc7640 1 -- 172.21.15.122:0/949548964 >> 172.21.15.122:0/949548964 conn(0x7fa6100b5d10 msgr2=0x7fa6100b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:10.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.268+0000 7fa615dc7640 1 -- 172.21.15.122:0/949548964 shutdown_connections 2024-03-25T18:26:10.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.268+0000 7fa615dc7640 1 -- 172.21.15.122:0/949548964 wait complete. 2024-03-25T18:26:10.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.269+0000 7fa615dc7640 10 monclient: build_initial_monmap 2024-03-25T18:26:10.271 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.269+0000 7fa615dc7640 10 monclient: monmap: 2024-03-25T18:26:10.271 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:26:10.271 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:10.271 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:26:10.272 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:26:10.272 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:10.272 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:26:10.272 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:26:10.272 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:26:10.272 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:26:10.272 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:10.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.270+0000 7fa615dc7640 1 librados: starting msgr at 2024-03-25T18:26:10.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.270+0000 7fa615dc7640 1 librados: starting objecter 2024-03-25T18:26:10.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.270+0000 7fa615dc7640 1 Processor -- start 2024-03-25T18:26:10.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.270+0000 7fa615dc7640 1 -- start start 2024-03-25T18:26:10.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.270+0000 7fa615dc7640 1 librados: setting wanted keys 2024-03-25T18:26:10.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.270+0000 7fa615dc7640 1 librados: calling monclient init 2024-03-25T18:26:10.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.270+0000 7fa615dc7640 10 monclient: init 2024-03-25T18:26:10.273 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.271+0000 7fa615dc7640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:10.274 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.271+0000 7fa615dc7640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:10.274 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.271+0000 7fa615dc7640 10 monclient(hunting): picked mon.noname-a con 0x7fa6100705b0 addr v1:172.21.15.122:6789/0 2024-03-25T18:26:10.274 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.271+0000 7fa615dc7640 10 monclient(hunting): picked mon.noname-b con 0x7fa6101402d0 addr v1:172.21.15.204:6789/0 2024-03-25T18:26:10.274 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.271+0000 7fa615dc7640 10 monclient(hunting): picked mon.noname-c con 0x7fa610149ae0 addr v1:172.21.15.122:6790/0 2024-03-25T18:26:10.274 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.271+0000 7fa615dc7640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6101442a0 con 0x7fa6100705b0 2024-03-25T18:26:10.274 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.271+0000 7fa615dc7640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6101d5520 con 0x7fa6101402d0 2024-03-25T18:26:10.274 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.271+0000 7fa615dc7640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa610142f90 con 0x7fa610149ae0 2024-03-25T18:26:10.274 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.271+0000 7fa615dc7640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:10.274 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.271+0000 7fa615dc7640 10 monclient(hunting): authenticate will time out at 7893.536538s 2024-03-25T18:26:10.285 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.285+0000 7fa607fff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fa6101402d0 0x7fa6101d5a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:35108/0 (socket says 172.21.15.122:35108) 2024-03-25T18:26:10.285 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.285+0000 7fa607fff640 1 -- 172.21.15.122:0/3594954495 learned_addr learned my addr 172.21.15.122:0/3594954495 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:26:10.293 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.292+0000 7fa60cff9640 10 client.?.objecter ms_handle_connect 0x7fa6101402d0 2024-03-25T18:26:10.293 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.292+0000 7fa60cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:10.303 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.302+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967682169 0 0) 0x7fa6101d5520 con 0x7fa6101402d0 2024-03-25T18:26:10.303 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.302+0000 7fa60cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:10.303 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.302+0000 7fa60cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:10.303 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.302+0000 7fa60cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:10.303 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.302+0000 7fa60cff9640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:26:10.303 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.302+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5f0004eb0 con 0x7fa6101402d0 2024-03-25T18:26:10.303 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.303+0000 7fa60cff9640 10 client.?.objecter ms_handle_connect 0x7fa6100705b0 2024-03-25T18:26:10.303 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.303+0000 7fa60cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:10.307 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.307+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2481518692 0 0) 0x7fa6101442a0 con 0x7fa6100705b0 2024-03-25T18:26:10.307 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.307+0000 7fa60cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:10.307 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.307+0000 7fa60cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:10.307 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.307+0000 7fa60cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:10.307 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.307+0000 7fa60cff9640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:26:10.307 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.307+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6101d5520 con 0x7fa6100705b0 2024-03-25T18:26:10.307 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.307+0000 7fa60cff9640 10 client.?.objecter ms_handle_connect 0x7fa610149ae0 2024-03-25T18:26:10.307 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.307+0000 7fa60cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.313+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1925583989 0 0) 0x7fa6101d5520 con 0x7fa6100705b0 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.313+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6101442a0 con 0x7fa6100705b0 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.313+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fa600001b00 con 0x7fa6100705b0 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.313+0000 7fa60cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.313+0000 7fa60cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.313+0000 7fa60cff9640 10 monclient(hunting): dump: 2024-03-25T18:26:10.314 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.313+0000 7fa60cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.313+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3457891886 0 0) 0x7fa5f0004eb0 con 0x7fa6101402d0 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.313+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6101d5520 con 0x7fa6101402d0 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.313+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fa5f8001a30 con 0x7fa6101402d0 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.313+0000 7fa60cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:10.315 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.313+0000 7fa60cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:10.316 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.313+0000 7fa60cff9640 10 monclient(hunting): dump: 2024-03-25T18:26:10.316 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:10.316 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:10.316 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:10.316 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:10.316 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:10.316 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:10.316 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:10.316 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:10.316 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:10.316 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:10.317 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.316+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800527745 0 0) 0x7fa610142f90 con 0x7fa610149ae0 2024-03-25T18:26:10.317 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.316+0000 7fa60cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:10.317 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.316+0000 7fa60cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:10.317 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.316+0000 7fa60cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:10.317 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.316+0000 7fa60cff9640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:26:10.317 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.316+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5f0004eb0 con 0x7fa610149ae0 2024-03-25T18:26:10.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1945497964 0 0) 0x7fa6101d5520 con 0x7fa6101402d0 2024-03-25T18:26:10.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 >> v1:172.21.15.122:6790/0 conn(0x7fa610149ae0 legacy=0x7fa610148d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.327 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 >> v1:172.21.15.122:6789/0 conn(0x7fa6100705b0 legacy=0x7fa610142880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa60cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:10.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa60cff9640 1 monclient: found mon.b 2024-03-25T18:26:10.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa60cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:10.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa60cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:26:10.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa610143120 con 0x7fa6101402d0 2024-03-25T18:26:10.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa60cff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:10.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa60cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:10.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa60cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:10.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa615dc7640 5 monclient: authenticate success, global_id 4663 2024-03-25T18:26:10.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa615dc7640 10 monclient: _renew_subs 2024-03-25T18:26:10.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa615dc7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:26:10.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa615dc7640 1 -- 172.21.15.122:0/3594954495 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa61013dfd0 con 0x7fa6101402d0 2024-03-25T18:26:10.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa615dc7640 10 client.4663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:10.328 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa615dc7640 10 monclient: _renew_subs 2024-03-25T18:26:10.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa615dc7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:26:10.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa615dc7640 1 -- 172.21.15.122:0/3594954495 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6101d6330 con 0x7fa6101402d0 2024-03-25T18:26:10.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.327+0000 7fa615dc7640 1 librados: init done 2024-03-25T18:26:10.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.328+0000 7fa615dc7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:10.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.328+0000 7fa615dc7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:10.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.328+0000 7fa615dc7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:26:10.329 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.328+0000 7fa615dc7640 1 -- 172.21.15.122:0/3594954495 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6100627b0 con 0x7fa6101402d0 2024-03-25T18:26:10.610 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.609+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa5f8001c70 con 0x7fa6101402d0 2024-03-25T18:26:10.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.609+0000 7fa60cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:10.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.609+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fa5f8003a60 con 0x7fa6101402d0 2024-03-25T18:26:10.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.609+0000 7fa60cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:10.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.609+0000 7fa60cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:10.621 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.609+0000 7fa60cff9640 10 monclient: dump: 2024-03-25T18:26:10.621 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:10.622 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:10.622 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:10.622 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:10.622 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:10.622 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:10.622 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:10.622 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:10.622 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:10.622 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:10.622 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.609+0000 7fa60e7fc640 4 set_mon_vals no callback set 2024-03-25T18:26:10.623 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.609+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fa5f8003c40 con 0x7fa6101402d0 2024-03-25T18:26:10.623 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.610+0000 7fa60f7fe640 10 monclient: get_auth_request con 0x7fa5f006df50 auth_method 0 2024-03-25T18:26:10.623 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.610+0000 7fa60cff9640 10 client.4663.objecter ms_handle_connect 0x7fa5f006df50 2024-03-25T18:26:10.623 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.614+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fa5f80548d0 con 0x7fa6101402d0 2024-03-25T18:26:10.623 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.614+0000 7fa60cff9640 10 client.4663.objecter ms_dispatch 0x7fa610000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:10.623 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.614+0000 7fa60cff9640 3 client.4663.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:10.624 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.614+0000 7fa60cff9640 3 client.4663.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:10.624 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.614+0000 7fa60cff9640 20 client.4663.objecter dump_active .. 0 homeless 2024-03-25T18:26:10.624 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.615+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa5f80503e0 con 0x7fa6101402d0 2024-03-25T18:26:10.624 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.615+0000 7fa60cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:10.624 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.615+0000 7fa60cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:10.838 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.836+0000 7fa615dc7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:10.838 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.836+0000 7fa615dc7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:10.838 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.836+0000 7fa615dc7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:26:10.838 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.836+0000 7fa615dc7640 1 -- 172.21.15.122:0/3594954495 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6101d56b0 con 0x7fa6101402d0 2024-03-25T18:26:10.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.974+0000 7fa60cff9640 1 -- 172.21.15.122:0/3594954495 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fa5f8052310 con 0x7fa6101402d0 2024-03-25T18:26:10.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.974+0000 7fa60cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:10.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.974+0000 7fa60cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:10.976 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:26:10.976 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:10.978 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.976+0000 7fa615dc7640 10 librados: watch_flush enter 2024-03-25T18:26:10.978 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.976+0000 7fa615dc7640 10 librados: watch_flush exit 2024-03-25T18:26:10.978 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.976+0000 7fa615dc7640 20 client.4663.objecter shutdown clearing up homeless session... 2024-03-25T18:26:10.978 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.976+0000 7fa615dc7640 10 client.4663.objecter successfully canceled tick 2024-03-25T18:26:10.978 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.976+0000 7fa615dc7640 1 -- 172.21.15.122:0/3594954495 >> v1:172.21.15.204:6800/676228005 conn(0x7fa5f006df50 legacy=0x7fa5f0070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.978 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.976+0000 7fa615dc7640 10 monclient: shutdown 2024-03-25T18:26:10.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.976+0000 7fa615dc7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:10.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.976+0000 7fa615dc7640 1 -- 172.21.15.122:0/3594954495 >> v1:172.21.15.204:6789/0 conn(0x7fa6101402d0 legacy=0x7fa6101d5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:10.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.976+0000 7fa615dc7640 1 -- 172.21.15.122:0/3594954495 shutdown_connections 2024-03-25T18:26:10.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.976+0000 7fa615dc7640 1 -- 172.21.15.122:0/3594954495 >> 172.21.15.122:0/3594954495 conn(0x7fa6100b5d10 msgr2=0x7fa6100b4d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:10.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.976+0000 7fa615dc7640 1 -- 172.21.15.122:0/3594954495 shutdown_connections 2024-03-25T18:26:10.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.976+0000 7fa615dc7640 1 -- 172.21.15.122:0/3594954495 wait complete. 2024-03-25T18:26:10.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:10.976+0000 7fa615dc7640 1 librados: shutdown 2024-03-25T18:26:10.990 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:12.630 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:10:06.372 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:12.683 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:10:06.425 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:12.725 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:10:06.467 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:12.725 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:10:06.467 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:12.735 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:10:06.481 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:12.742 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:10:06.488 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:12.744 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:10:06.487 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:12.771 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:10:06.516 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:12.777 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:10:06.520 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:17.797 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:17.915 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.913+0000 7f90274fb640 10 monclient: get_monmap_and_config 2024-03-25T18:26:17.915 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.913+0000 7f90274fb640 10 monclient: build_initial_monmap 2024-03-25T18:26:17.915 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.913+0000 7f90274fb640 10 monclient: monmap: 2024-03-25T18:26:17.915 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:26:17.915 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:17.915 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:26:17.914719+0000 2024-03-25T18:26:17.915 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:26:17.914719+0000 2024-03-25T18:26:17.915 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:17.915 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:17.915 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:26:17.915 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:26:17.915 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:26:17.916 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:17.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.914+0000 7f90274fb640 1 Processor -- start 2024-03-25T18:26:17.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.914+0000 7f90274fb640 1 -- start start 2024-03-25T18:26:17.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.914+0000 7f90274fb640 10 monclient: init 2024-03-25T18:26:17.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.915+0000 7f90274fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:17.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.915+0000 7f90274fb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:17.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.915+0000 7f90274fb640 10 monclient(hunting): picked mon.noname-a con 0x7f90200a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:26:17.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.915+0000 7f90274fb640 10 monclient(hunting): picked mon.noname-c con 0x7f902013f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:26:17.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.915+0000 7f90274fb640 10 monclient(hunting): picked mon.noname-b con 0x7f90201430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:26:17.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.915+0000 7f90274fb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9020142520 con 0x7f90200a98f0 2024-03-25T18:26:17.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.915+0000 7f90274fb640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90200b28a0 con 0x7f902013f710 2024-03-25T18:26:17.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.915+0000 7f90274fb640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90200b2a30 con 0x7f90201430c0 2024-03-25T18:26:17.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.915+0000 7f90274fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:17.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.915+0000 7f90274fb640 10 monclient(hunting): authenticate will time out at 7901.181487s 2024-03-25T18:26:17.922 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.921+0000 7f9025297640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f90200a98f0 0x7f90200a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:59756/0 (socket says 172.21.15.122:59756) 2024-03-25T18:26:17.922 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.921+0000 7f9025297640 1 -- 172.21.15.122:0/2869363577 learned_addr learned my addr 172.21.15.122:0/2869363577 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:26:17.939 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.937+0000 7f9017fff640 1 -- 172.21.15.122:0/2869363577 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630351748 0 0) 0x7f90200b28a0 con 0x7f902013f710 2024-03-25T18:26:17.939 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.937+0000 7f9017fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:17.939 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.937+0000 7f9017fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:17.939 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.937+0000 7f9017fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:17.939 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.937+0000 7f9017fff640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:26:17.939 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.937+0000 7f9017fff640 1 -- 172.21.15.122:0/2869363577 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f900c003520 con 0x7f902013f710 2024-03-25T18:26:17.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.946+0000 7f9017fff640 1 -- 172.21.15.122:0/2869363577 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3400793248 0 0) 0x7f90200b2a30 con 0x7f90201430c0 2024-03-25T18:26:17.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.946+0000 7f9017fff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:17.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.946+0000 7f9017fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:17.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.946+0000 7f9017fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:17.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.946+0000 7f9017fff640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:26:17.947 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.946+0000 7f9017fff640 1 -- 172.21.15.122:0/2869363577 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90200b28a0 con 0x7f90201430c0 2024-03-25T18:26:17.958 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.957+0000 7f9017fff640 1 -- 172.21.15.122:0/2869363577 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3562895487 0 0) 0x7f900c003520 con 0x7f902013f710 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.957+0000 7f9017fff640 1 -- 172.21.15.122:0/2869363577 >> v1:172.21.15.122:6790/0 conn(0x7f90201430c0 legacy=0x7f902014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.957+0000 7f9017fff640 1 -- 172.21.15.122:0/2869363577 >> v1:172.21.15.122:6789/0 conn(0x7f90200a98f0 legacy=0x7f90200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.957+0000 7f9017fff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.957+0000 7f9017fff640 1 monclient: found mon.noname-c 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.957+0000 7f9017fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.957+0000 7f9017fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.957+0000 7f9017fff640 1 -- 172.21.15.122:0/2869363577 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90200b44c0 con 0x7f902013f710 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.957+0000 7f9017fff640 10 monclient: _finish_auth 0 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.957+0000 7f9017fff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.957+0000 7f9017fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.957+0000 7f9017fff640 1 -- 172.21.15.122:0/2869363577 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9010001690 con 0x7f902013f710 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.957+0000 7f9017fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.957+0000 7f9017fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.957+0000 7f9017fff640 10 monclient: dump: 2024-03-25T18:26:17.959 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:17.960 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:17.960 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:17.960 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:17.960 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:17.960 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:17.960 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:17.960 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:17.960 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:17.960 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:17.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.957+0000 7f90274fb640 5 monclient: authenticate success, global_id 4666 2024-03-25T18:26:17.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.957+0000 7f90274fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:17.983 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.981+0000 7f9017fff640 1 -- 172.21.15.122:0/2869363577 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9010001870 con 0x7f902013f710 2024-03-25T18:26:17.983 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.981+0000 7f9017fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:17.983 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.981+0000 7f9017fff640 1 -- 172.21.15.122:0/2869363577 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9010001c80 con 0x7f902013f710 2024-03-25T18:26:17.983 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.981+0000 7f90274fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:17.983 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.981+0000 7f90274fb640 4 set_mon_vals no callback set 2024-03-25T18:26:17.983 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.981+0000 7f9017fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:17.983 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.981+0000 7f9017fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:17.983 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.981+0000 7f9017fff640 10 monclient: dump: 2024-03-25T18:26:17.983 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:17.983 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:17.984 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:17.984 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:17.984 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:17.984 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:17.984 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:17.984 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:17.984 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:17.984 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:17.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.982+0000 7f90274fb640 10 monclient: shutdown 2024-03-25T18:26:17.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.982+0000 7f90274fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:17.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.982+0000 7f90274fb640 1 -- 172.21.15.122:0/2869363577 >> v1:172.21.15.204:6789/0 conn(0x7f902013f710 legacy=0x7f90201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:17.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.982+0000 7f90274fb640 1 -- 172.21.15.122:0/2869363577 shutdown_connections 2024-03-25T18:26:17.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.982+0000 7f90274fb640 1 -- 172.21.15.122:0/2869363577 >> 172.21.15.122:0/2869363577 conn(0x7f90200b7ea0 msgr2=0x7f90200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:17.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.983+0000 7f90274fb640 1 -- 172.21.15.122:0/2869363577 shutdown_connections 2024-03-25T18:26:17.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.983+0000 7f90274fb640 1 -- 172.21.15.122:0/2869363577 wait complete. 2024-03-25T18:26:17.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.983+0000 7f90274fb640 10 monclient: build_initial_monmap 2024-03-25T18:26:17.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.983+0000 7f90274fb640 10 monclient: monmap: 2024-03-25T18:26:17.987 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:26:17.987 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:17.987 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:26:17.987 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:26:17.987 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:17.987 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:26:17.987 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:26:17.987 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:26:17.987 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:26:17.987 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:17.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.984+0000 7f90274fb640 1 librados: starting msgr at 2024-03-25T18:26:17.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.984+0000 7f90274fb640 1 librados: starting objecter 2024-03-25T18:26:17.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.984+0000 7f90274fb640 1 Processor -- start 2024-03-25T18:26:17.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.984+0000 7f90274fb640 1 -- start start 2024-03-25T18:26:17.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.984+0000 7f90274fb640 1 librados: setting wanted keys 2024-03-25T18:26:17.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.984+0000 7f90274fb640 1 librados: calling monclient init 2024-03-25T18:26:17.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.984+0000 7f90274fb640 10 monclient: init 2024-03-25T18:26:17.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.984+0000 7f90274fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:17.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.984+0000 7f90274fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:17.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.984+0000 7f90274fb640 10 monclient(hunting): picked mon.noname-a con 0x7f90200a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:26:17.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.984+0000 7f90274fb640 10 monclient(hunting): picked mon.noname-c con 0x7f902013f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:26:17.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.985+0000 7f90274fb640 10 monclient(hunting): picked mon.noname-b con 0x7f90201430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:26:17.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.985+0000 7f90274fb640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90201354c0 con 0x7f90200a98f0 2024-03-25T18:26:17.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.985+0000 7f90274fb640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90201319b0 con 0x7f90201430c0 2024-03-25T18:26:17.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.985+0000 7f90274fb640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9020135070 con 0x7f902013f710 2024-03-25T18:26:17.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.985+0000 7f90274fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:17.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.985+0000 7f90274fb640 10 monclient(hunting): authenticate will time out at 7901.251025s 2024-03-25T18:26:17.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.991+0000 7f9024a96640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f902013f710 0x7f9020130b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:53906/0 (socket says 172.21.15.122:53906) 2024-03-25T18:26:17.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.991+0000 7f9024a96640 1 -- 172.21.15.122:0/3778923118 learned_addr learned my addr 172.21.15.122:0/3778923118 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:26:17.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.991+0000 7f9015ffb640 10 client.?.objecter ms_handle_connect 0x7f902013f710 2024-03-25T18:26:17.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:17.991+0000 7f9015ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:18.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.004+0000 7f9015ffb640 10 client.?.objecter ms_handle_connect 0x7f90201430c0 2024-03-25T18:26:18.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.004+0000 7f9015ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:18.006 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.004+0000 7f9015ffb640 10 client.?.objecter ms_handle_connect 0x7f90200a98f0 2024-03-25T18:26:18.006 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.004+0000 7f9015ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:18.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.007+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3854452024 0 0) 0x7f9020135070 con 0x7f902013f710 2024-03-25T18:26:18.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.007+0000 7f9015ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:18.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.007+0000 7f9015ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:18.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.007+0000 7f9015ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:18.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.007+0000 7f9015ffb640 10 monclient(hunting): my global_id is 4637 2024-03-25T18:26:18.008 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.007+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9004004eb0 con 0x7f902013f710 2024-03-25T18:26:18.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.012+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847402958 0 0) 0x7f90201354c0 con 0x7f90200a98f0 2024-03-25T18:26:18.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.012+0000 7f9015ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:18.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.012+0000 7f9015ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:18.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.012+0000 7f9015ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:18.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.012+0000 7f9015ffb640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:26:18.013 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.012+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9020135070 con 0x7f90200a98f0 2024-03-25T18:26:18.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.016+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 777838031 0 0) 0x7f90201319b0 con 0x7f90201430c0 2024-03-25T18:26:18.017 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.016+0000 7f9015ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:26:18.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.016+0000 7f9015ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:18.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.016+0000 7f9015ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:18.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.016+0000 7f9015ffb640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:26:18.018 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.016+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90201354c0 con 0x7f90201430c0 2024-03-25T18:26:18.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.022+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3808165213 0 0) 0x7f9004004eb0 con 0x7f902013f710 2024-03-25T18:26:18.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.022+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f90201319b0 con 0x7f902013f710 2024-03-25T18:26:18.025 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.022+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9010001c40 con 0x7f902013f710 2024-03-25T18:26:18.025 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.022+0000 7f9015ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:18.025 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.022+0000 7f9015ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:18.025 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.022+0000 7f9015ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:18.025 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:18.025 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:18.025 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:18.025 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:18.025 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:18.025 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:18.025 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:18.025 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:18.025 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:18.025 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:18.026 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.023+0000 7f9015ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:18.026 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.024+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 21465185 0 0) 0x7f90201354c0 con 0x7f90201430c0 2024-03-25T18:26:18.026 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.024+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9004004eb0 con 0x7f90201430c0 2024-03-25T18:26:18.028 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.027+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9018001b90 con 0x7f90201430c0 2024-03-25T18:26:18.028 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.027+0000 7f9015ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:18.028 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.027+0000 7f9015ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:18.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.027+0000 7f9015ffb640 10 monclient(hunting): dump: 2024-03-25T18:26:18.029 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:18.029 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:18.029 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:18.029 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:18.029 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:18.029 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:18.029 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:18.029 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:18.029 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:18.029 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:18.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.032+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3506043452 0 0) 0x7f90201319b0 con 0x7f902013f710 2024-03-25T18:26:18.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.032+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 >> v1:172.21.15.204:6789/0 conn(0x7f90201430c0 legacy=0x7f90201312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:18.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.032+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 >> v1:172.21.15.122:6789/0 conn(0x7f90200a98f0 legacy=0x7f90201d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:18.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.032+0000 7f9015ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:26:18.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.032+0000 7f9015ffb640 1 monclient: found mon.c 2024-03-25T18:26:18.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.032+0000 7f9015ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:18.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.032+0000 7f9015ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:18.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.032+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9020135200 con 0x7f902013f710 2024-03-25T18:26:18.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.032+0000 7f9015ffb640 10 monclient: _finish_auth 0 2024-03-25T18:26:18.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.032+0000 7f9015ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:26:18.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.032+0000 7f9015ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:18.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.032+0000 7f90274fb640 5 monclient: authenticate success, global_id 4637 2024-03-25T18:26:18.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.032+0000 7f90274fb640 10 monclient: _renew_subs 2024-03-25T18:26:18.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.032+0000 7f90274fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:18.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.032+0000 7f90274fb640 1 -- 172.21.15.122:0/3778923118 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90200b7010 con 0x7f902013f710 2024-03-25T18:26:18.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.033+0000 7f90274fb640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:18.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.033+0000 7f90274fb640 10 monclient: _renew_subs 2024-03-25T18:26:18.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.033+0000 7f90274fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:18.035 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.033+0000 7f90274fb640 1 -- 172.21.15.122:0/3778923118 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f902013ba70 con 0x7f902013f710 2024-03-25T18:26:18.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.033+0000 7f90274fb640 1 librados: init done 2024-03-25T18:26:18.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.034+0000 7f90274fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:18.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.034+0000 7f90274fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:18.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.034+0000 7f90274fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:18.036 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.034+0000 7f90274fb640 1 -- 172.21.15.122:0/3778923118 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90200b73b0 con 0x7f902013f710 2024-03-25T18:26:18.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.329+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f90100011f0 con 0x7f902013f710 2024-03-25T18:26:18.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.330+0000 7f9015ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:18.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.330+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f9010003b00 con 0x7f902013f710 2024-03-25T18:26:18.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.330+0000 7f9015ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:18.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.330+0000 7f9015ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:18.331 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.330+0000 7f9015ffb640 10 monclient: dump: 2024-03-25T18:26:18.332 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:18.332 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:18.332 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:18.332 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:18.332 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:18.332 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:18.332 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:18.332 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:18.332 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:18.332 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:18.332 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.330+0000 7f90177fe640 4 set_mon_vals no callback set 2024-03-25T18:26:18.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.330+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f9010003d20 con 0x7f902013f710 2024-03-25T18:26:18.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.331+0000 7f9025297640 10 monclient: get_auth_request con 0x7f900406dec0 auth_method 0 2024-03-25T18:26:18.333 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.331+0000 7f9015ffb640 10 client.4637.objecter ms_handle_connect 0x7f900406dec0 2024-03-25T18:26:18.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.337+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7f9010054930 con 0x7f902013f710 2024-03-25T18:26:18.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.337+0000 7f9015ffb640 10 client.4637.objecter ms_dispatch 0x7f9020000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:18.338 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.337+0000 7f9015ffb640 3 client.4637.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:18.339 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.337+0000 7f9015ffb640 3 client.4637.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:18.339 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.337+0000 7f9015ffb640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-25T18:26:18.339 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.337+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9010050440 con 0x7f902013f710 2024-03-25T18:26:18.339 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.337+0000 7f9015ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:18.339 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.337+0000 7f9015ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:18.591 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.590+0000 7f90274fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:18.591 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.590+0000 7f90274fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:18.591 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.590+0000 7f90274fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:18.591 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.590+0000 7f90274fb640 1 -- 172.21.15.122:0/3778923118 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9020135670 con 0x7f902013f710 2024-03-25T18:26:18.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.609+0000 7f9015ffb640 1 -- 172.21.15.122:0/3778923118 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7f9020135670 con 0x7f902013f710 2024-03-25T18:26:18.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.610+0000 7f9015ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:18.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.610+0000 7f9015ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:18.611 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:26:18.611 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:18.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.612+0000 7f90274fb640 10 librados: watch_flush enter 2024-03-25T18:26:18.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.612+0000 7f90274fb640 10 librados: watch_flush exit 2024-03-25T18:26:18.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.612+0000 7f90274fb640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-25T18:26:18.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.612+0000 7f90274fb640 10 client.4637.objecter successfully canceled tick 2024-03-25T18:26:18.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.612+0000 7f90274fb640 1 -- 172.21.15.122:0/3778923118 >> v1:172.21.15.204:6800/676228005 conn(0x7f900406dec0 legacy=0x7f9004070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:18.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.612+0000 7f90274fb640 10 monclient: shutdown 2024-03-25T18:26:18.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.612+0000 7f90274fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:18.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.612+0000 7f90274fb640 1 -- 172.21.15.122:0/3778923118 >> v1:172.21.15.122:6790/0 conn(0x7f902013f710 legacy=0x7f9020130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:18.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.612+0000 7f90274fb640 1 -- 172.21.15.122:0/3778923118 shutdown_connections 2024-03-25T18:26:18.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.612+0000 7f90274fb640 1 -- 172.21.15.122:0/3778923118 >> 172.21.15.122:0/3778923118 conn(0x7f90200b7ea0 msgr2=0x7f902013d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:18.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.612+0000 7f90274fb640 1 -- 172.21.15.122:0/3778923118 shutdown_connections 2024-03-25T18:26:18.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.613+0000 7f90274fb640 1 -- 172.21.15.122:0/3778923118 wait complete. 2024-03-25T18:26:18.615 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:18.613+0000 7f90274fb640 1 librados: shutdown 2024-03-25T18:26:18.626 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:25.433 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:25.545 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.543+0000 7fb870b10640 10 monclient: get_monmap_and_config 2024-03-25T18:26:25.545 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.543+0000 7fb870b10640 10 monclient: build_initial_monmap 2024-03-25T18:26:25.545 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.543+0000 7fb870b10640 10 monclient: monmap: 2024-03-25T18:26:25.545 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:26:25.545 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:25.545 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:26:25.545008+0000 2024-03-25T18:26:25.545 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:26:25.545008+0000 2024-03-25T18:26:25.545 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:25.545 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:25.545 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:26:25.546 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:26:25.546 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:26:25.546 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:25.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.544+0000 7fb870b10640 1 Processor -- start 2024-03-25T18:26:25.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.544+0000 7fb870b10640 1 -- start start 2024-03-25T18:26:25.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.544+0000 7fb870b10640 10 monclient: init 2024-03-25T18:26:25.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.545+0000 7fb870b10640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:25.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.545+0000 7fb870b10640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:26:25.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.545+0000 7fb870b10640 10 monclient(hunting): picked mon.noname-b con 0x7fb86c0a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:26:25.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.545+0000 7fb870b10640 10 monclient(hunting): picked mon.noname-c con 0x7fb86c13f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:26:25.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.545+0000 7fb870b10640 10 monclient(hunting): picked mon.noname-a con 0x7fb86c1430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:26:25.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.545+0000 7fb870b10640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c142520 con 0x7fb86c1430c0 2024-03-25T18:26:25.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.545+0000 7fb870b10640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c0b28a0 con 0x7fb86c13f710 2024-03-25T18:26:25.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.545+0000 7fb870b10640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c0b2a30 con 0x7fb86c0a98f0 2024-03-25T18:26:25.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.545+0000 7fb870b10640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:25.548 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.545+0000 7fb870b10640 10 monclient(hunting): authenticate will time out at 7908.811544s 2024-03-25T18:26:25.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.553+0000 7fb86a59c640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7fb86c0a98f0 0x7fb86c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:41746/0 (socket says 172.21.15.122:41746) 2024-03-25T18:26:25.555 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.553+0000 7fb86a59c640 1 -- 172.21.15.122:0/3083811483 learned_addr learned my addr 172.21.15.122:0/3083811483 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:26:25.566 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.562+0000 7fb869d9b640 1 -- 172.21.15.122:0/3083811483 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1513196355 0 0) 0x7fb86c0b2a30 con 0x7fb86c0a98f0 2024-03-25T18:26:25.566 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.562+0000 7fb869d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:26:25.566 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.562+0000 7fb869d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:25.566 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.562+0000 7fb869d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:25.566 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.562+0000 7fb869d9b640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:26:25.566 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.562+0000 7fb869d9b640 1 -- 172.21.15.122:0/3083811483 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb858003520 con 0x7fb86c0a98f0 2024-03-25T18:26:25.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.571+0000 7fb869d9b640 1 -- 172.21.15.122:0/3083811483 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1380519766 0 0) 0x7fb858003520 con 0x7fb86c0a98f0 2024-03-25T18:26:25.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.571+0000 7fb869d9b640 1 -- 172.21.15.122:0/3083811483 >> v1:172.21.15.204:6789/0 conn(0x7fb86c13f710 legacy=0x7fb86c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:25.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.571+0000 7fb869d9b640 1 -- 172.21.15.122:0/3083811483 >> v1:172.21.15.122:6789/0 conn(0x7fb86c1430c0 legacy=0x7fb86c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:25.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.571+0000 7fb869d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:26:25.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.571+0000 7fb869d9b640 1 monclient: found mon.noname-b 2024-03-25T18:26:25.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.571+0000 7fb869d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:25.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.571+0000 7fb869d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:26:25.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.571+0000 7fb869d9b640 1 -- 172.21.15.122:0/3083811483 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c0b44c0 con 0x7fb86c0a98f0 2024-03-25T18:26:25.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.571+0000 7fb869d9b640 10 monclient: _finish_auth 0 2024-03-25T18:26:25.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.571+0000 7fb869d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:26:25.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.571+0000 7fb869d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:25.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.571+0000 7fb869d9b640 1 -- 172.21.15.122:0/3083811483 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fb85c001690 con 0x7fb86c0a98f0 2024-03-25T18:26:25.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.571+0000 7fb869d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:25.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.571+0000 7fb869d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:25.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.571+0000 7fb869d9b640 10 monclient: dump: 2024-03-25T18:26:25.574 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:25.574 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:25.574 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:25.574 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:25.574 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:25.574 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:25.574 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:25.575 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:25.575 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:25.575 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:25.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.571+0000 7fb870b10640 5 monclient: authenticate success, global_id 4640 2024-03-25T18:26:25.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.571+0000 7fb870b10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:25.582 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.581+0000 7fb869d9b640 1 -- 172.21.15.122:0/3083811483 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb85c0017f0 con 0x7fb86c0a98f0 2024-03-25T18:26:25.582 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.581+0000 7fb869d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:25.582 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.581+0000 7fb869d9b640 1 -- 172.21.15.122:0/3083811483 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fb85c001c00 con 0x7fb86c0a98f0 2024-03-25T18:26:25.582 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.581+0000 7fb869d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:25.582 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.581+0000 7fb869d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:25.583 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.581+0000 7fb869d9b640 10 monclient: dump: 2024-03-25T18:26:25.583 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:25.583 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:25.583 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:25.583 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:25.583 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:25.583 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:25.583 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:25.583 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:25.583 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:25.583 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:25.583 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.581+0000 7fb870b10640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:25.583 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.581+0000 7fb870b10640 4 set_mon_vals no callback set 2024-03-25T18:26:25.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.581+0000 7fb870b10640 10 monclient: shutdown 2024-03-25T18:26:25.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.581+0000 7fb870b10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:25.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.581+0000 7fb870b10640 1 -- 172.21.15.122:0/3083811483 >> v1:172.21.15.122:6790/0 conn(0x7fb86c0a98f0 legacy=0x7fb86c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:25.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.582+0000 7fb870b10640 1 -- 172.21.15.122:0/3083811483 shutdown_connections 2024-03-25T18:26:25.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.582+0000 7fb870b10640 1 -- 172.21.15.122:0/3083811483 >> 172.21.15.122:0/3083811483 conn(0x7fb86c0b7ea0 msgr2=0x7fb86c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:25.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.582+0000 7fb870b10640 1 -- 172.21.15.122:0/3083811483 shutdown_connections 2024-03-25T18:26:25.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.582+0000 7fb870b10640 1 -- 172.21.15.122:0/3083811483 wait complete. 2024-03-25T18:26:25.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.583+0000 7fb870b10640 10 monclient: build_initial_monmap 2024-03-25T18:26:25.586 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.583+0000 7fb870b10640 10 monclient: monmap: 2024-03-25T18:26:25.586 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:26:25.586 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:25.587 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:26:25.587 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:26:25.587 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:25.587 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:26:25.588 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:26:25.588 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:26:25.588 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:26:25.588 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:25.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.583+0000 7fb870b10640 1 librados: starting msgr at 2024-03-25T18:26:25.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.583+0000 7fb870b10640 1 librados: starting objecter 2024-03-25T18:26:25.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.583+0000 7fb870b10640 1 Processor -- start 2024-03-25T18:26:25.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.583+0000 7fb870b10640 1 -- start start 2024-03-25T18:26:25.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.583+0000 7fb870b10640 1 librados: setting wanted keys 2024-03-25T18:26:25.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.583+0000 7fb870b10640 1 librados: calling monclient init 2024-03-25T18:26:25.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.583+0000 7fb870b10640 10 monclient: init 2024-03-25T18:26:25.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.584+0000 7fb870b10640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:25.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.584+0000 7fb870b10640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:25.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.584+0000 7fb870b10640 10 monclient(hunting): picked mon.noname-a con 0x7fb86c0a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:26:25.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.584+0000 7fb870b10640 10 monclient(hunting): picked mon.noname-c con 0x7fb86c13f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:26:25.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.584+0000 7fb870b10640 10 monclient(hunting): picked mon.noname-b con 0x7fb86c1430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:26:25.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.584+0000 7fb870b10640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c135730 con 0x7fb86c0a98f0 2024-03-25T18:26:25.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.584+0000 7fb870b10640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c1358c0 con 0x7fb86c1430c0 2024-03-25T18:26:25.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.584+0000 7fb870b10640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c135a50 con 0x7fb86c13f710 2024-03-25T18:26:25.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.584+0000 7fb870b10640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:25.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.584+0000 7fb870b10640 10 monclient(hunting): authenticate will time out at 7908.850697s 2024-03-25T18:26:25.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.592+0000 7fb863fff640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7fb86c13f710 0x7fb86c1d41e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:41796/0 (socket says 172.21.15.122:41796) 2024-03-25T18:26:25.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.592+0000 7fb863fff640 1 -- 172.21.15.122:0/2006055248 learned_addr learned my addr 172.21.15.122:0/2006055248 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:26:25.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.593+0000 7fb862ffd640 10 client.?.objecter ms_handle_connect 0x7fb86c13f710 2024-03-25T18:26:25.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.593+0000 7fb862ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:25.602 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.601+0000 7fb862ffd640 10 client.?.objecter ms_handle_connect 0x7fb86c1430c0 2024-03-25T18:26:25.602 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.601+0000 7fb862ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:25.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.607+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1849583517 0 0) 0x7fb86c135a50 con 0x7fb86c13f710 2024-03-25T18:26:25.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.607+0000 7fb862ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:25.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.607+0000 7fb862ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:25.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.607+0000 7fb862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:25.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.607+0000 7fb862ffd640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:26:25.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.607+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb84c004eb0 con 0x7fb86c13f710 2024-03-25T18:26:25.618 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.617+0000 7fb862ffd640 10 client.?.objecter ms_handle_connect 0x7fb86c0a98f0 2024-03-25T18:26:25.618 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.617+0000 7fb862ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:25.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.625+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3171515341 0 0) 0x7fb86c1358c0 con 0x7fb86c1430c0 2024-03-25T18:26:25.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.625+0000 7fb862ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:25.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.625+0000 7fb862ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:25.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.625+0000 7fb862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:25.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.625+0000 7fb862ffd640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:26:25.627 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.625+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb86c135a50 con 0x7fb86c1430c0 2024-03-25T18:26:25.628 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.626+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1306714536 0 0) 0x7fb84c004eb0 con 0x7fb86c13f710 2024-03-25T18:26:25.628 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.626+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb86c1358c0 con 0x7fb86c13f710 2024-03-25T18:26:25.628 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.626+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fb854001a30 con 0x7fb86c13f710 2024-03-25T18:26:25.628 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.626+0000 7fb862ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:25.628 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.626+0000 7fb862ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:25.628 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.626+0000 7fb862ffd640 10 monclient(hunting): dump: 2024-03-25T18:26:25.628 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:25.628 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:25.629 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:25.629 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:25.629 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:25.629 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:25.629 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:25.629 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:25.629 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:25.629 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:25.629 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.626+0000 7fb862ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:25.630 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.629+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426203686 0 0) 0x7fb86c135730 con 0x7fb86c0a98f0 2024-03-25T18:26:25.630 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.629+0000 7fb862ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:25.630 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.629+0000 7fb862ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:25.630 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.629+0000 7fb862ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:25.630 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.629+0000 7fb862ffd640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:26:25.630 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.629+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb84c004eb0 con 0x7fb86c0a98f0 2024-03-25T18:26:25.638 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.636+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1336221958 0 0) 0x7fb86c135a50 con 0x7fb86c1430c0 2024-03-25T18:26:25.638 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.637+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb86c135730 con 0x7fb86c1430c0 2024-03-25T18:26:25.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1232856881 0 0) 0x7fb86c1358c0 con 0x7fb86c13f710 2024-03-25T18:26:25.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 >> v1:172.21.15.204:6789/0 conn(0x7fb86c1430c0 legacy=0x7fb86c1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:25.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 >> v1:172.21.15.122:6789/0 conn(0x7fb86c0a98f0 legacy=0x7fb86c1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:25.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb862ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:25.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb862ffd640 1 monclient: found mon.c 2024-03-25T18:26:25.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb862ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:25.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb862ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:25.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c1d5000 con 0x7fb86c13f710 2024-03-25T18:26:25.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb862ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:25.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb862ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:25.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb862ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:25.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb870b10640 5 monclient: authenticate success, global_id 4643 2024-03-25T18:26:25.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb870b10640 10 monclient: _renew_subs 2024-03-25T18:26:25.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb870b10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:25.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb870b10640 1 -- 172.21.15.122:0/2006055248 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86c1d52b0 con 0x7fb86c13f710 2024-03-25T18:26:25.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb870b10640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:25.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb870b10640 10 monclient: _renew_subs 2024-03-25T18:26:25.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb870b10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:25.644 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb870b10640 1 -- 172.21.15.122:0/2006055248 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb86c13c900 con 0x7fb86c13f710 2024-03-25T18:26:25.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.641+0000 7fb870b10640 1 librados: init done 2024-03-25T18:26:25.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.642+0000 7fb870b10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:25.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.642+0000 7fb870b10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:25.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.642+0000 7fb870b10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:25.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.642+0000 7fb870b10640 1 -- 172.21.15.122:0/2006055248 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb86c13cbb0 con 0x7fb86c13f710 2024-03-25T18:26:25.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.929+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb854001c70 con 0x7fb86c13f710 2024-03-25T18:26:25.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.929+0000 7fb862ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:25.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.929+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fb8540038c0 con 0x7fb86c13f710 2024-03-25T18:26:25.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.929+0000 7fb862ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:25.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.929+0000 7fb862ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:25.931 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.929+0000 7fb862ffd640 10 monclient: dump: 2024-03-25T18:26:25.932 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:25.932 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:25.932 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:25.932 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:25.932 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:25.932 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:25.932 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:25.932 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:25.932 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:25.932 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:25.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.929+0000 7fb86959a640 4 set_mon_vals no callback set 2024-03-25T18:26:25.932 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.930+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fb854003ae0 con 0x7fb86c13f710 2024-03-25T18:26:25.933 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.930+0000 7fb86a59c640 10 monclient: get_auth_request con 0x7fb84c06e0d0 auth_method 0 2024-03-25T18:26:25.933 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.931+0000 7fb862ffd640 10 client.4643.objecter ms_handle_connect 0x7fb84c06e0d0 2024-03-25T18:26:25.938 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.936+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 3477610916 0 0) 0x7fb854054680 con 0x7fb86c13f710 2024-03-25T18:26:25.938 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.936+0000 7fb862ffd640 10 client.4643.objecter ms_dispatch 0x7fb86c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:26:25.938 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.936+0000 7fb862ffd640 3 client.4643.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:26:25.938 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.936+0000 7fb862ffd640 3 client.4643.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:26:25.939 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.937+0000 7fb862ffd640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-25T18:26:25.939 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.937+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb854050190 con 0x7fb86c13f710 2024-03-25T18:26:25.939 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.937+0000 7fb862ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:25.939 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:25.937+0000 7fb862ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:26.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.188+0000 7fb870b10640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:26.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.188+0000 7fb870b10640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:26.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.188+0000 7fb870b10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:26.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.188+0000 7fb870b10640 1 -- 172.21.15.122:0/2006055248 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb86c1371c0 con 0x7fb86c13f710 2024-03-25T18:26:26.221 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.220+0000 7fb862ffd640 1 -- 172.21.15.122:0/2006055248 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 3082987142) 0x7fb8540520c0 con 0x7fb86c13f710 2024-03-25T18:26:26.221 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.220+0000 7fb862ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:26.222 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.220+0000 7fb862ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:26.222 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:26:26.222 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":10,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:16:34.161248+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:16:34.161248+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:26.224 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.222+0000 7fb870b10640 10 librados: watch_flush enter 2024-03-25T18:26:26.224 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.222+0000 7fb870b10640 10 librados: watch_flush exit 2024-03-25T18:26:26.224 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.222+0000 7fb870b10640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-25T18:26:26.224 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.222+0000 7fb870b10640 10 client.4643.objecter successfully canceled tick 2024-03-25T18:26:26.224 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.222+0000 7fb870b10640 1 -- 172.21.15.122:0/2006055248 >> v1:172.21.15.204:6800/676228005 conn(0x7fb84c06e0d0 legacy=0x7fb84c070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:26.224 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.222+0000 7fb870b10640 10 monclient: shutdown 2024-03-25T18:26:26.224 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.222+0000 7fb870b10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:26.224 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.222+0000 7fb870b10640 1 -- 172.21.15.122:0/2006055248 >> v1:172.21.15.122:6790/0 conn(0x7fb86c13f710 legacy=0x7fb86c1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:26.224 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.222+0000 7fb870b10640 1 -- 172.21.15.122:0/2006055248 shutdown_connections 2024-03-25T18:26:26.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.222+0000 7fb870b10640 1 -- 172.21.15.122:0/2006055248 >> 172.21.15.122:0/2006055248 conn(0x7fb86c0b7ea0 msgr2=0x7fb86c14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:26.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.222+0000 7fb870b10640 1 -- 172.21.15.122:0/2006055248 shutdown_connections 2024-03-25T18:26:26.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.222+0000 7fb870b10640 1 -- 172.21.15.122:0/2006055248 wait complete. 2024-03-25T18:26:26.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:26.223+0000 7fb870b10640 1 librados: shutdown 2024-03-25T18:26:26.235 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:32.911 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:10:26.653 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:32.916 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:10:26.658 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:32.923 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:10:26.669 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:32.925 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:10:26.670 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:32.933 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:10:26.676 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:32.980 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:10:26.722 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:32.989 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:10:26.734 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:32.998 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:10:26.740 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.023 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:10:26.765 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:33.042 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:33.136 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.134+0000 7f2959257640 10 monclient: get_monmap_and_config 2024-03-25T18:26:33.136 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.134+0000 7f2959257640 10 monclient: build_initial_monmap 2024-03-25T18:26:33.136 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.134+0000 7f2959257640 10 monclient: monmap: 2024-03-25T18:26:33.136 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:26:33.136 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:33.136 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:26:33.135969+0000 2024-03-25T18:26:33.136 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:26:33.135969+0000 2024-03-25T18:26:33.136 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:33.136 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:33.136 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:26:33.137 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:26:33.137 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:26:33.137 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:33.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.135+0000 7f2959257640 1 Processor -- start 2024-03-25T18:26:33.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.135+0000 7f2959257640 1 -- start start 2024-03-25T18:26:33.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.135+0000 7f2959257640 10 monclient: init 2024-03-25T18:26:33.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.136+0000 7f2959257640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:33.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.136+0000 7f2959257640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:33.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.136+0000 7f2959257640 10 monclient(hunting): picked mon.noname-a con 0x7f29540a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:26:33.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.136+0000 7f2959257640 10 monclient(hunting): picked mon.noname-c con 0x7f295413f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:26:33.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.136+0000 7f2959257640 10 monclient(hunting): picked mon.noname-b con 0x7f29541430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:26:33.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.136+0000 7f2959257640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2954142520 con 0x7f29540a98f0 2024-03-25T18:26:33.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.136+0000 7f2959257640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29540b28a0 con 0x7f295413f710 2024-03-25T18:26:33.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.136+0000 7f2959257640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29540b2a30 con 0x7f29541430c0 2024-03-25T18:26:33.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.136+0000 7f2959257640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:33.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.136+0000 7f2959257640 10 monclient(hunting): authenticate will time out at 7916.402513s 2024-03-25T18:26:33.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.137+0000 7f294a59c640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f295413f710 0x7f29541429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:48494/0 (socket says 172.21.15.122:48494) 2024-03-25T18:26:33.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.137+0000 7f294a59c640 1 -- 172.21.15.122:0/3770715596 learned_addr learned my addr 172.21.15.122:0/3770715596 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:26:33.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.162+0000 7f295259c640 1 -- 172.21.15.122:0/3770715596 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1865128258 0 0) 0x7f2954142520 con 0x7f29540a98f0 2024-03-25T18:26:33.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.162+0000 7f295259c640 10 monclient(hunting): _negotiate 2024-03-25T18:26:33.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.162+0000 7f295259c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:33.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.162+0000 7f295259c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:33.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.162+0000 7f295259c640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:26:33.163 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.162+0000 7f295259c640 1 -- 172.21.15.122:0/3770715596 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2940003520 con 0x7f29540a98f0 2024-03-25T18:26:33.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f295259c640 1 -- 172.21.15.122:0/3770715596 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3673896588 0 0) 0x7f2940003520 con 0x7f29540a98f0 2024-03-25T18:26:33.176 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f295259c640 1 -- 172.21.15.122:0/3770715596 >> v1:172.21.15.122:6790/0 conn(0x7f29541430c0 legacy=0x7f295414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:33.177 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f295259c640 1 -- 172.21.15.122:0/3770715596 >> v1:172.21.15.204:6789/0 conn(0x7f295413f710 legacy=0x7f29541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:33.177 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f295259c640 10 monclient: _finish_hunting 0 2024-03-25T18:26:33.177 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f295259c640 1 monclient: found mon.noname-a 2024-03-25T18:26:33.177 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f295259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:33.177 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f295259c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:26:33.177 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f295259c640 1 -- 172.21.15.122:0/3770715596 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29540b44c0 con 0x7f29540a98f0 2024-03-25T18:26:33.177 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f295259c640 10 monclient: _finish_auth 0 2024-03-25T18:26:33.177 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f295259c640 10 monclient: _check_auth_tickets 2024-03-25T18:26:33.178 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f295259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:33.178 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f295259c640 1 -- 172.21.15.122:0/3770715596 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f2944001690 con 0x7f29540a98f0 2024-03-25T18:26:33.178 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f2959257640 5 monclient: authenticate success, global_id 4689 2024-03-25T18:26:33.178 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f2959257640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:33.178 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f295259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:33.178 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f295259c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:33.178 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f295259c640 10 monclient: dump: 2024-03-25T18:26:33.179 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:33.179 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:33.179 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:33.179 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:33.179 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:33.179 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:33.179 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:33.180 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:33.180 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:33.180 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:33.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.174+0000 7f2959257640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:33.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.182+0000 7f295259c640 1 -- 172.21.15.122:0/3770715596 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29440017f0 con 0x7f29540a98f0 2024-03-25T18:26:33.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.182+0000 7f295259c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:33.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.182+0000 7f295259c640 1 -- 172.21.15.122:0/3770715596 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f2944001c00 con 0x7f29540a98f0 2024-03-25T18:26:33.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.182+0000 7f2959257640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:33.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.182+0000 7f2959257640 4 set_mon_vals no callback set 2024-03-25T18:26:33.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.182+0000 7f295259c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:33.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.182+0000 7f295259c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:33.184 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.182+0000 7f295259c640 10 monclient: dump: 2024-03-25T18:26:33.184 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:33.185 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:33.185 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:33.185 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:33.185 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:33.185 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:33.185 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:33.185 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:33.185 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:33.185 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:33.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.183+0000 7f2959257640 10 monclient: shutdown 2024-03-25T18:26:33.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.183+0000 7f2959257640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:33.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.183+0000 7f2959257640 1 -- 172.21.15.122:0/3770715596 >> v1:172.21.15.122:6789/0 conn(0x7f29540a98f0 legacy=0x7f29540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:33.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.183+0000 7f2959257640 1 -- 172.21.15.122:0/3770715596 shutdown_connections 2024-03-25T18:26:33.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.183+0000 7f2959257640 1 -- 172.21.15.122:0/3770715596 >> 172.21.15.122:0/3770715596 conn(0x7f29540b7ea0 msgr2=0x7f29540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:33.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.183+0000 7f2959257640 1 -- 172.21.15.122:0/3770715596 shutdown_connections 2024-03-25T18:26:33.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.183+0000 7f2959257640 1 -- 172.21.15.122:0/3770715596 wait complete. 2024-03-25T18:26:33.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.184+0000 7f2959257640 10 monclient: build_initial_monmap 2024-03-25T18:26:33.188 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.184+0000 7f2959257640 10 monclient: monmap: 2024-03-25T18:26:33.188 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:26:33.188 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:33.188 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:26:33.188 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:26:33.189 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:33.189 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:26:33.189 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:26:33.189 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:26:33.189 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:26:33.189 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:33.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.184+0000 7f2959257640 1 librados: starting msgr at 2024-03-25T18:26:33.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.184+0000 7f2959257640 1 librados: starting objecter 2024-03-25T18:26:33.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.184+0000 7f2959257640 1 Processor -- start 2024-03-25T18:26:33.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.185+0000 7f2959257640 1 -- start start 2024-03-25T18:26:33.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.185+0000 7f2959257640 1 librados: setting wanted keys 2024-03-25T18:26:33.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.185+0000 7f2959257640 1 librados: calling monclient init 2024-03-25T18:26:33.189 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.185+0000 7f2959257640 10 monclient: init 2024-03-25T18:26:33.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.185+0000 7f2959257640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:33.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.185+0000 7f2959257640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:26:33.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.185+0000 7f2959257640 10 monclient(hunting): picked mon.noname-a con 0x7f29540a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:26:33.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.185+0000 7f2959257640 10 monclient(hunting): picked mon.noname-b con 0x7f295413f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:26:33.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.185+0000 7f2959257640 10 monclient(hunting): picked mon.noname-c con 0x7f29541430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:26:33.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.185+0000 7f2959257640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29540abd30 con 0x7f29540a98f0 2024-03-25T18:26:33.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.185+0000 7f2959257640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29541d77a0 con 0x7f295413f710 2024-03-25T18:26:33.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.185+0000 7f2959257640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29541d51f0 con 0x7f29541430c0 2024-03-25T18:26:33.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.185+0000 7f2959257640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:33.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.185+0000 7f2959257640 10 monclient(hunting): authenticate will time out at 7916.451573s 2024-03-25T18:26:33.190 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.186+0000 7f295359e640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f29541430c0 0x7f29541e1ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:40940/0 (socket says 172.21.15.122:40940) 2024-03-25T18:26:33.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.186+0000 7f295359e640 1 -- 172.21.15.122:0/1190996105 learned_addr learned my addr 172.21.15.122:0/1190996105 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:26:33.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.186+0000 7f294bfff640 10 client.?.objecter ms_handle_connect 0x7f295413f710 2024-03-25T18:26:33.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.186+0000 7f294bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:33.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.187+0000 7f294bfff640 10 client.?.objecter ms_handle_connect 0x7f29541430c0 2024-03-25T18:26:33.191 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.187+0000 7f294bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:33.249 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.248+0000 7f294bfff640 10 client.?.objecter ms_handle_connect 0x7f29540a98f0 2024-03-25T18:26:33.250 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.248+0000 7f294bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:33.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.265+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 250451684 0 0) 0x7f29540abd30 con 0x7f29540a98f0 2024-03-25T18:26:33.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.265+0000 7f294bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:33.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.265+0000 7f294bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:33.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.265+0000 7f294bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:33.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.265+0000 7f294bfff640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:26:33.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.265+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2934004eb0 con 0x7f29540a98f0 2024-03-25T18:26:33.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.270+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833465815 0 0) 0x7f29541d51f0 con 0x7f29541430c0 2024-03-25T18:26:33.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.270+0000 7f294bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:33.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.270+0000 7f294bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:33.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.270+0000 7f294bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:33.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.270+0000 7f294bfff640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:26:33.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.270+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29540abd30 con 0x7f29541430c0 2024-03-25T18:26:33.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.278+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 422949165 0 0) 0x7f2934004eb0 con 0x7f29540a98f0 2024-03-25T18:26:33.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.278+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29541d51f0 con 0x7f29540a98f0 2024-03-25T18:26:33.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.278+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f2944001bc0 con 0x7f29540a98f0 2024-03-25T18:26:33.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.278+0000 7f294bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:33.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.278+0000 7f294bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:33.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.278+0000 7f294bfff640 10 monclient(hunting): dump: 2024-03-25T18:26:33.280 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:33.280 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:33.280 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:33.280 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:33.280 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:33.280 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:33.280 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:33.280 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:33.280 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:33.280 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:33.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.278+0000 7f294bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:33.281 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.278+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3778862340 0 0) 0x7f29541d77a0 con 0x7f295413f710 2024-03-25T18:26:33.281 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.278+0000 7f294bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:26:33.281 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.278+0000 7f294bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:33.281 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.278+0000 7f294bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:33.281 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.278+0000 7f294bfff640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:26:33.281 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.278+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2934004eb0 con 0x7f295413f710 2024-03-25T18:26:33.288 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.286+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2837898136 0 0) 0x7f2934004eb0 con 0x7f295413f710 2024-03-25T18:26:33.288 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.287+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29541d77a0 con 0x7f295413f710 2024-03-25T18:26:33.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.288+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f293c0018c0 con 0x7f295413f710 2024-03-25T18:26:33.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.288+0000 7f294bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:33.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.288+0000 7f294bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:33.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.288+0000 7f294bfff640 10 monclient(hunting): dump: 2024-03-25T18:26:33.289 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:33.289 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:33.289 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:33.289 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:33.289 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:33.289 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:33.289 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:33.290 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:33.290 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:33.290 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:33.291 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 269753909 0 0) 0x7f29541d51f0 con 0x7f29540a98f0 2024-03-25T18:26:33.291 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 >> v1:172.21.15.122:6790/0 conn(0x7f29541430c0 legacy=0x7f29541e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:33.291 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 >> v1:172.21.15.204:6789/0 conn(0x7f295413f710 legacy=0x7f2954138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:33.291 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f294bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:26:33.291 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f294bfff640 1 monclient: found mon.a 2024-03-25T18:26:33.291 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f294bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:33.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f294bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:26:33.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29541d83d0 con 0x7f29540a98f0 2024-03-25T18:26:33.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f294bfff640 10 monclient: _finish_auth 0 2024-03-25T18:26:33.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f294bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:26:33.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f294bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:33.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f2959257640 5 monclient: authenticate success, global_id 4692 2024-03-25T18:26:33.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f2959257640 10 monclient: _renew_subs 2024-03-25T18:26:33.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f2959257640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:26:33.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f2959257640 1 -- 172.21.15.122:0/1190996105 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2954138a50 con 0x7f29540a98f0 2024-03-25T18:26:33.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f2959257640 10 client.4692.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:33.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f2959257640 10 monclient: _renew_subs 2024-03-25T18:26:33.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f2959257640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:26:33.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f2959257640 1 -- 172.21.15.122:0/1190996105 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29541e35c0 con 0x7f29540a98f0 2024-03-25T18:26:33.292 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.289+0000 7f2959257640 1 librados: init done 2024-03-25T18:26:33.293 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.290+0000 7f2959257640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:33.293 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.290+0000 7f2959257640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:33.293 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.290+0000 7f2959257640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:26:33.293 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.290+0000 7f2959257640 1 -- 172.21.15.122:0/1190996105 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29540627b0 con 0x7f29540a98f0 2024-03-25T18:26:33.301 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.300+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29440038a0 con 0x7f29540a98f0 2024-03-25T18:26:33.301 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.300+0000 7f294bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:33.301 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.300+0000 7f2951d9b640 4 set_mon_vals no callback set 2024-03-25T18:26:33.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.607+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f2944003bb0 con 0x7f29540a98f0 2024-03-25T18:26:33.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.607+0000 7f294bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:33.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.607+0000 7f294bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:33.609 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.607+0000 7f294bfff640 10 monclient: dump: 2024-03-25T18:26:33.609 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:33.609 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:33.609 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:33.609 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:33.609 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:33.609 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:33.609 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:33.609 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:33.610 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:33.610 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:33.610 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.608+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f2944019b80 con 0x7f29540a98f0 2024-03-25T18:26:33.610 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.609+0000 7f294a59c640 10 monclient: get_auth_request con 0x7f293406df30 auth_method 0 2024-03-25T18:26:33.610 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.609+0000 7f294bfff640 10 client.4692.objecter ms_handle_connect 0x7f293406df30 2024-03-25T18:26:33.616 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.614+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1633957047 0 0) 0x7f2944053730 con 0x7f29540a98f0 2024-03-25T18:26:33.616 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.614+0000 7f294bfff640 10 client.4692.objecter ms_dispatch 0x7f2954000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:26:33.616 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.614+0000 7f294bfff640 3 client.4692.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:26:33.616 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.614+0000 7f294bfff640 3 client.4692.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:26:33.616 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.614+0000 7f294bfff640 20 client.4692.objecter dump_active .. 0 homeless 2024-03-25T18:26:33.616 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.614+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f294404f070 con 0x7f29540a98f0 2024-03-25T18:26:33.616 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.614+0000 7f294bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:33.616 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.614+0000 7f294bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:33.875 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.874+0000 7f2959257640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:33.875 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.874+0000 7f2959257640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:33.875 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.874+0000 7f2959257640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:26:33.875 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.874+0000 7f2959257640 1 -- 172.21.15.122:0/1190996105 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2954074cc0 con 0x7f29540a98f0 2024-03-25T18:26:33.906 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.904+0000 7f294bfff640 1 -- 172.21.15.122:0/1190996105 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1879996311) 0x7f2944051170 con 0x7f29540a98f0 2024-03-25T18:26:33.906 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.904+0000 7f294bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:33.906 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.904+0000 7f294bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:33.906 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:26:33.906 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":11,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:26:32.649240+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:32.649240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:33.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.906+0000 7f2959257640 10 librados: watch_flush enter 2024-03-25T18:26:33.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.906+0000 7f2959257640 10 librados: watch_flush exit 2024-03-25T18:26:33.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.906+0000 7f2959257640 20 client.4692.objecter shutdown clearing up homeless session... 2024-03-25T18:26:33.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.906+0000 7f2959257640 10 client.4692.objecter successfully canceled tick 2024-03-25T18:26:33.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.906+0000 7f2959257640 1 -- 172.21.15.122:0/1190996105 >> v1:172.21.15.204:6800/676228005 conn(0x7f293406df30 legacy=0x7f29340703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:33.908 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.906+0000 7f2959257640 10 monclient: shutdown 2024-03-25T18:26:33.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.906+0000 7f2959257640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:33.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.906+0000 7f2959257640 1 -- 172.21.15.122:0/1190996105 >> v1:172.21.15.122:6789/0 conn(0x7f29540a98f0 legacy=0x7f29541d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:33.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.906+0000 7f2959257640 1 -- 172.21.15.122:0/1190996105 shutdown_connections 2024-03-25T18:26:33.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.906+0000 7f2959257640 1 -- 172.21.15.122:0/1190996105 >> 172.21.15.122:0/1190996105 conn(0x7f29540b7ea0 msgr2=0x7f2954074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:33.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.907+0000 7f2959257640 1 -- 172.21.15.122:0/1190996105 shutdown_connections 2024-03-25T18:26:33.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.907+0000 7f2959257640 1 -- 172.21.15.122:0/1190996105 wait complete. 2024-03-25T18:26:33.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:33.907+0000 7f2959257640 1 librados: shutdown 2024-03-25T18:26:33.921 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:40.728 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:40.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.823+0000 7ffb9a806640 10 monclient: get_monmap_and_config 2024-03-25T18:26:40.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.823+0000 7ffb9a806640 10 monclient: build_initial_monmap 2024-03-25T18:26:40.825 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.823+0000 7ffb9a806640 10 monclient: monmap: 2024-03-25T18:26:40.825 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:26:40.826 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:40.826 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:26:40.825053+0000 2024-03-25T18:26:40.826 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:26:40.825053+0000 2024-03-25T18:26:40.826 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:40.826 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:40.826 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:26:40.826 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:26:40.826 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:26:40.826 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:40.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.824+0000 7ffb9a806640 1 Processor -- start 2024-03-25T18:26:40.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.824+0000 7ffb9a806640 1 -- start start 2024-03-25T18:26:40.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.824+0000 7ffb9a806640 10 monclient: init 2024-03-25T18:26:40.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.825+0000 7ffb9a806640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:40.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.825+0000 7ffb9a806640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:26:40.827 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.825+0000 7ffb9a806640 10 monclient(hunting): picked mon.noname-a con 0x7ffb940a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:26:40.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.825+0000 7ffb9a806640 10 monclient(hunting): picked mon.noname-b con 0x7ffb9413f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:26:40.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.825+0000 7ffb9a806640 10 monclient(hunting): picked mon.noname-c con 0x7ffb941430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:26:40.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.825+0000 7ffb9a806640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb94142520 con 0x7ffb940a98f0 2024-03-25T18:26:40.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.825+0000 7ffb9a806640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb940b28a0 con 0x7ffb941430c0 2024-03-25T18:26:40.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.825+0000 7ffb9a806640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb940b2a30 con 0x7ffb9413f710 2024-03-25T18:26:40.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.825+0000 7ffb9a806640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:40.828 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.825+0000 7ffb9a806640 10 monclient(hunting): authenticate will time out at 7924.091440s 2024-03-25T18:26:40.840 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.839+0000 7ffb93fff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7ffb940a98f0 0x7ffb940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:60694/0 (socket says 172.21.15.122:60694) 2024-03-25T18:26:40.840 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.839+0000 7ffb93fff640 1 -- 172.21.15.122:0/1445204594 learned_addr learned my addr 172.21.15.122:0/1445204594 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:26:40.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.851+0000 7ffb92ffd640 1 -- 172.21.15.122:0/1445204594 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291147195 0 0) 0x7ffb94142520 con 0x7ffb940a98f0 2024-03-25T18:26:40.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.851+0000 7ffb92ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.851+0000 7ffb92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.851+0000 7ffb92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.851+0000 7ffb92ffd640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:26:40.852 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.851+0000 7ffb92ffd640 1 -- 172.21.15.122:0/1445204594 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb7c003520 con 0x7ffb940a98f0 2024-03-25T18:26:40.863 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb92ffd640 1 -- 172.21.15.122:0/1445204594 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1000361886 0 0) 0x7ffb7c003520 con 0x7ffb940a98f0 2024-03-25T18:26:40.863 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb92ffd640 1 -- 172.21.15.122:0/1445204594 >> v1:172.21.15.122:6790/0 conn(0x7ffb9413f710 legacy=0x7ffb941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:26:40.863 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb92ffd640 1 -- 172.21.15.122:0/1445204594 >> v1:172.21.15.204:6789/0 conn(0x7ffb941430c0 legacy=0x7ffb9414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:26:40.863 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb92ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:40.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb92ffd640 1 monclient: found mon.noname-a 2024-03-25T18:26:40.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:40.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb92ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:26:40.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb92ffd640 1 -- 172.21.15.122:0/1445204594 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb940b44c0 con 0x7ffb940a98f0 2024-03-25T18:26:40.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb92ffd640 10 monclient: _finish_auth 0 2024-03-25T18:26:40.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb92ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:40.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:40.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb92ffd640 1 -- 172.21.15.122:0/1445204594 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ffb84001690 con 0x7ffb940a98f0 2024-03-25T18:26:40.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb9a806640 5 monclient: authenticate success, global_id 4695 2024-03-25T18:26:40.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb9a806640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:40.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb92ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:26:40.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb92ffd640 10 monclient: dump: 2024-03-25T18:26:40.864 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:40.865 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:40.865 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:40.865 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:40.865 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.865 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:40.865 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:40.865 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:40.865 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:40.865 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:40.865 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.862+0000 7ffb9a806640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:40.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.888+0000 7ffb92ffd640 1 -- 172.21.15.122:0/1445204594 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb84001860 con 0x7ffb940a98f0 2024-03-25T18:26:40.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.888+0000 7ffb92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:40.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.888+0000 7ffb92ffd640 1 -- 172.21.15.122:0/1445204594 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ffb84001cb0 con 0x7ffb940a98f0 2024-03-25T18:26:40.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.888+0000 7ffb92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.888+0000 7ffb92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:26:40.889 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.888+0000 7ffb92ffd640 10 monclient: dump: 2024-03-25T18:26:40.890 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:40.890 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:40.890 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:40.890 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:40.890 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.890 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:40.890 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:40.890 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:40.890 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:40.890 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:40.890 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.888+0000 7ffb9a806640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:40.890 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.888+0000 7ffb9a806640 4 set_mon_vals no callback set 2024-03-25T18:26:40.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.888+0000 7ffb9a806640 10 monclient: shutdown 2024-03-25T18:26:40.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.888+0000 7ffb9a806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:40.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.888+0000 7ffb9a806640 1 -- 172.21.15.122:0/1445204594 >> v1:172.21.15.122:6789/0 conn(0x7ffb940a98f0 legacy=0x7ffb940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.888+0000 7ffb9a806640 1 -- 172.21.15.122:0/1445204594 shutdown_connections 2024-03-25T18:26:40.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.888+0000 7ffb9a806640 1 -- 172.21.15.122:0/1445204594 >> 172.21.15.122:0/1445204594 conn(0x7ffb940b7ea0 msgr2=0x7ffb940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:40.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.888+0000 7ffb9a806640 1 -- 172.21.15.122:0/1445204594 shutdown_connections 2024-03-25T18:26:40.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.889+0000 7ffb9a806640 1 -- 172.21.15.122:0/1445204594 wait complete. 2024-03-25T18:26:40.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.889+0000 7ffb9a806640 10 monclient: build_initial_monmap 2024-03-25T18:26:40.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.889+0000 7ffb9a806640 10 monclient: monmap: 2024-03-25T18:26:40.893 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:26:40.893 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:40.893 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:26:40.893 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:26:40.893 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:40.893 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:26:40.893 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:26:40.893 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:26:40.893 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:26:40.893 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:40.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.890+0000 7ffb9a806640 1 librados: starting msgr at 2024-03-25T18:26:40.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.890+0000 7ffb9a806640 1 librados: starting objecter 2024-03-25T18:26:40.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.890+0000 7ffb9a806640 1 Processor -- start 2024-03-25T18:26:40.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.890+0000 7ffb9a806640 1 -- start start 2024-03-25T18:26:40.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.890+0000 7ffb9a806640 1 librados: setting wanted keys 2024-03-25T18:26:40.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.890+0000 7ffb9a806640 1 librados: calling monclient init 2024-03-25T18:26:40.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.890+0000 7ffb9a806640 10 monclient: init 2024-03-25T18:26:40.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.891+0000 7ffb9a806640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:40.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.891+0000 7ffb9a806640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:40.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.891+0000 7ffb9a806640 10 monclient(hunting): picked mon.noname-b con 0x7ffb940a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:26:40.894 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.891+0000 7ffb9a806640 10 monclient(hunting): picked mon.noname-c con 0x7ffb9413f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:26:40.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.891+0000 7ffb9a806640 10 monclient(hunting): picked mon.noname-a con 0x7ffb941430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:26:40.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.891+0000 7ffb9a806640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb941d76e0 con 0x7ffb941430c0 2024-03-25T18:26:40.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.891+0000 7ffb9a806640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb940abd80 con 0x7ffb940a98f0 2024-03-25T18:26:40.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.891+0000 7ffb9a806640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb94137c90 con 0x7ffb9413f710 2024-03-25T18:26:40.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.891+0000 7ffb9a806640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:40.895 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.891+0000 7ffb9a806640 10 monclient(hunting): authenticate will time out at 7924.157487s 2024-03-25T18:26:40.898 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.897+0000 7ffb937fe640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7ffb9413f710 0x7ffb94138670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:44840/0 (socket says 172.21.15.122:44840) 2024-03-25T18:26:40.898 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.897+0000 7ffb937fe640 1 -- 172.21.15.122:0/2810278009 learned_addr learned my addr 172.21.15.122:0/2810278009 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:26:40.899 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.897+0000 7ffb90ff9640 10 client.?.objecter ms_handle_connect 0x7ffb9413f710 2024-03-25T18:26:40.899 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.897+0000 7ffb90ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:40.936 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.935+0000 7ffb90ff9640 10 client.?.objecter ms_handle_connect 0x7ffb940a98f0 2024-03-25T18:26:40.936 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.935+0000 7ffb90ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:40.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.944+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2107590134 0 0) 0x7ffb940abd80 con 0x7ffb940a98f0 2024-03-25T18:26:40.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.944+0000 7ffb90ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.944+0000 7ffb90ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.944+0000 7ffb90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.944+0000 7ffb90ff9640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:26:40.945 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.944+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb80004eb0 con 0x7ffb940a98f0 2024-03-25T18:26:40.958 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.957+0000 7ffb90ff9640 10 client.?.objecter ms_handle_connect 0x7ffb941430c0 2024-03-25T18:26:40.958 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.957+0000 7ffb90ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:40.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.971+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984155053 0 0) 0x7ffb941d76e0 con 0x7ffb941430c0 2024-03-25T18:26:40.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.971+0000 7ffb90ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.971+0000 7ffb90ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.971+0000 7ffb90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.971+0000 7ffb90ff9640 10 monclient(hunting): my global_id is 4698 2024-03-25T18:26:40.972 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.971+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb940abd80 con 0x7ffb941430c0 2024-03-25T18:26:40.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.974+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39335759 0 0) 0x7ffb94137c90 con 0x7ffb9413f710 2024-03-25T18:26:40.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.974+0000 7ffb90ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:40.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.974+0000 7ffb90ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:40.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.974+0000 7ffb90ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:40.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.974+0000 7ffb90ff9640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:26:40.975 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.974+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb941d76e0 con 0x7ffb9413f710 2024-03-25T18:26:40.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.977+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3423327264 0 0) 0x7ffb80004eb0 con 0x7ffb940a98f0 2024-03-25T18:26:40.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.977+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb94137c90 con 0x7ffb940a98f0 2024-03-25T18:26:40.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.977+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ffb84001b70 con 0x7ffb940a98f0 2024-03-25T18:26:40.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.977+0000 7ffb90ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:40.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.977+0000 7ffb90ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:40.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.977+0000 7ffb90ff9640 10 monclient(hunting): dump: 2024-03-25T18:26:40.979 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:40.979 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:40.979 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:40.980 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:40.980 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:40.980 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:40.980 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:40.980 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:40.980 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:40.980 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:40.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.978+0000 7ffb90ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:40.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1067142918 0 0) 0x7ffb94137c90 con 0x7ffb940a98f0 2024-03-25T18:26:40.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 >> v1:172.21.15.122:6790/0 conn(0x7ffb9413f710 legacy=0x7ffb94138670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 >> v1:172.21.15.122:6789/0 conn(0x7ffb941430c0 legacy=0x7ffb941e2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:40.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb90ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:40.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb90ff9640 1 monclient: found mon.b 2024-03-25T18:26:40.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb90ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:40.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb90ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:26:40.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb941d41e0 con 0x7ffb940a98f0 2024-03-25T18:26:40.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb90ff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:40.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb90ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:40.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb90ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:40.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb9a806640 5 monclient: authenticate success, global_id 4681 2024-03-25T18:26:40.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb9a806640 10 monclient: _renew_subs 2024-03-25T18:26:40.986 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb9a806640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:26:40.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb9a806640 1 -- 172.21.15.122:0/2810278009 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb941d82d0 con 0x7ffb940a98f0 2024-03-25T18:26:40.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb9a806640 10 client.4681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:40.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb9a806640 10 monclient: _renew_subs 2024-03-25T18:26:40.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb9a806640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:26:40.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb9a806640 1 -- 172.21.15.122:0/2810278009 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb94138d80 con 0x7ffb940a98f0 2024-03-25T18:26:40.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.984+0000 7ffb9a806640 1 librados: init done 2024-03-25T18:26:40.987 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.985+0000 7ffb9a806640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:40.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.985+0000 7ffb9a806640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:40.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.985+0000 7ffb9a806640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:26:40.988 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:40.985+0000 7ffb9a806640 1 -- 172.21.15.122:0/2810278009 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb940627b0 con 0x7ffb940a98f0 2024-03-25T18:26:41.297 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.296+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb84001d70 con 0x7ffb940a98f0 2024-03-25T18:26:41.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.296+0000 7ffb90ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:41.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.296+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ffb84004d90 con 0x7ffb940a98f0 2024-03-25T18:26:41.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.296+0000 7ffb90ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:41.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.296+0000 7ffb90ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:41.298 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.296+0000 7ffb90ff9640 10 monclient: dump: 2024-03-25T18:26:41.298 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:41.298 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:41.298 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:41.298 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:41.298 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:41.298 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:41.298 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:41.298 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:41.298 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:41.299 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:41.299 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.296+0000 7ffb927fc640 4 set_mon_vals no callback set 2024-03-25T18:26:41.299 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.297+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7ffb8401ad60 con 0x7ffb940a98f0 2024-03-25T18:26:41.299 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.297+0000 7ffb937fe640 10 monclient: get_auth_request con 0x7ffb8006dda0 auth_method 0 2024-03-25T18:26:41.299 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.298+0000 7ffb90ff9640 10 client.4681.objecter ms_handle_connect 0x7ffb8006dda0 2024-03-25T18:26:41.304 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.303+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1633957047 0 0) 0x7ffb840548f0 con 0x7ffb940a98f0 2024-03-25T18:26:41.304 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.303+0000 7ffb90ff9640 10 client.4681.objecter ms_dispatch 0x7ffb94000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:26:41.304 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.303+0000 7ffb90ff9640 3 client.4681.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:26:41.304 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.303+0000 7ffb90ff9640 3 client.4681.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:26:41.304 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.303+0000 7ffb90ff9640 20 client.4681.objecter dump_active .. 0 homeless 2024-03-25T18:26:41.305 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.303+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb84050230 con 0x7ffb940a98f0 2024-03-25T18:26:41.305 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.303+0000 7ffb90ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:41.305 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.303+0000 7ffb90ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:41.558 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.556+0000 7ffb9a806640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:41.558 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.556+0000 7ffb9a806640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:41.558 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.556+0000 7ffb9a806640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:26:41.558 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.556+0000 7ffb9a806640 1 -- 172.21.15.122:0/2810278009 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb9413b430 con 0x7ffb940a98f0 2024-03-25T18:26:41.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.603+0000 7ffb90ff9640 1 -- 172.21.15.122:0/2810278009 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1879996311) 0x7ffb84052330 con 0x7ffb940a98f0 2024-03-25T18:26:41.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.603+0000 7ffb90ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:41.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.603+0000 7ffb90ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:41.605 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:26:41.605 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":11,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:26:32.649240+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:32.649240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:41.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.606+0000 7ffb9a806640 10 librados: watch_flush enter 2024-03-25T18:26:41.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.606+0000 7ffb9a806640 10 librados: watch_flush exit 2024-03-25T18:26:41.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.606+0000 7ffb9a806640 20 client.4681.objecter shutdown clearing up homeless session... 2024-03-25T18:26:41.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.606+0000 7ffb9a806640 10 client.4681.objecter successfully canceled tick 2024-03-25T18:26:41.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.606+0000 7ffb9a806640 1 -- 172.21.15.122:0/2810278009 >> v1:172.21.15.204:6800/676228005 conn(0x7ffb8006dda0 legacy=0x7ffb80070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.606+0000 7ffb9a806640 10 monclient: shutdown 2024-03-25T18:26:41.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.606+0000 7ffb9a806640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:41.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.606+0000 7ffb9a806640 1 -- 172.21.15.122:0/2810278009 >> v1:172.21.15.204:6789/0 conn(0x7ffb940a98f0 legacy=0x7ffb941d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:41.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.606+0000 7ffb9a806640 1 -- 172.21.15.122:0/2810278009 shutdown_connections 2024-03-25T18:26:41.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.606+0000 7ffb9a806640 1 -- 172.21.15.122:0/2810278009 >> 172.21.15.122:0/2810278009 conn(0x7ffb940b7ea0 msgr2=0x7ffb94074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:41.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.606+0000 7ffb9a806640 1 -- 172.21.15.122:0/2810278009 shutdown_connections 2024-03-25T18:26:41.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.606+0000 7ffb9a806640 1 -- 172.21.15.122:0/2810278009 wait complete. 2024-03-25T18:26:41.608 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:41.606+0000 7ffb9a806640 1 librados: shutdown 2024-03-25T18:26:41.621 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:48.428 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:48.544 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.544+0000 7f53a808f640 10 monclient: get_monmap_and_config 2024-03-25T18:26:48.544 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.544+0000 7f53a808f640 10 monclient: build_initial_monmap 2024-03-25T18:26:48.545 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.544+0000 7f53a808f640 10 monclient: monmap: 2024-03-25T18:26:48.545 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:26:48.545 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:48.545 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:26:48.544423+0000 2024-03-25T18:26:48.545 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:26:48.544423+0000 2024-03-25T18:26:48.545 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:48.545 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:48.545 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:26:48.545 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:26:48.545 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:26:48.545 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:48.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.544+0000 7f53a808f640 1 Processor -- start 2024-03-25T18:26:48.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.545+0000 7f53a808f640 1 -- start start 2024-03-25T18:26:48.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.545+0000 7f53a808f640 10 monclient: init 2024-03-25T18:26:48.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.545+0000 7f53a808f640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:48.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.545+0000 7f53a808f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:26:48.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.546+0000 7f53a808f640 10 monclient(hunting): picked mon.noname-c con 0x7f53a00b4d60 addr v1:172.21.15.204:6789/0 2024-03-25T18:26:48.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.546+0000 7f53a808f640 10 monclient(hunting): picked mon.noname-b con 0x7f53a00b1730 addr v1:172.21.15.122:6790/0 2024-03-25T18:26:48.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.546+0000 7f53a808f640 10 monclient(hunting): picked mon.noname-a con 0x7f53a014be90 addr v1:172.21.15.122:6789/0 2024-03-25T18:26:48.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.546+0000 7f53a808f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a00af940 con 0x7f53a014be90 2024-03-25T18:26:48.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.546+0000 7f53a808f640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a00b0a70 con 0x7f53a00b4d60 2024-03-25T18:26:48.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.546+0000 7f53a808f640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a00b0d50 con 0x7f53a00b1730 2024-03-25T18:26:48.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.546+0000 7f53a808f640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:48.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.546+0000 7f53a808f640 10 monclient(hunting): authenticate will time out at 7931.811211s 2024-03-25T18:26:48.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.554+0000 7f53a562a640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f53a00b1730 0x7f53a00b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:49764/0 (socket says 172.21.15.122:49764) 2024-03-25T18:26:48.554 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.554+0000 7f53a562a640 1 -- 172.21.15.122:0/2392537511 learned_addr learned my addr 172.21.15.122:0/2392537511 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:26:48.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.573+0000 7f53a4e29640 1 -- 172.21.15.122:0/2392537511 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760244365 0 0) 0x7f53a00af940 con 0x7f53a014be90 2024-03-25T18:26:48.573 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.573+0000 7f53a4e29640 10 monclient(hunting): _negotiate 2024-03-25T18:26:48.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.573+0000 7f53a4e29640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:48.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.573+0000 7f53a4e29640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:48.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.573+0000 7f53a4e29640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:26:48.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.573+0000 7f53a4e29640 1 -- 172.21.15.122:0/2392537511 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f538c003520 con 0x7f53a014be90 2024-03-25T18:26:48.574 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.574+0000 7f53a4e29640 1 -- 172.21.15.122:0/2392537511 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323545868 0 0) 0x7f53a00b0d50 con 0x7f53a00b1730 2024-03-25T18:26:48.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.574+0000 7f53a4e29640 10 monclient(hunting): _negotiate 2024-03-25T18:26:48.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.574+0000 7f53a4e29640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:48.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.574+0000 7f53a4e29640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:48.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.574+0000 7f53a4e29640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:26:48.575 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.574+0000 7f53a4e29640 1 -- 172.21.15.122:0/2392537511 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a00af940 con 0x7f53a00b1730 2024-03-25T18:26:48.578 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.578+0000 7f53a4e29640 1 -- 172.21.15.122:0/2392537511 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051253182 0 0) 0x7f53a00b0a70 con 0x7f53a00b4d60 2024-03-25T18:26:48.578 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.578+0000 7f53a4e29640 10 monclient(hunting): _negotiate 2024-03-25T18:26:48.578 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.578+0000 7f53a4e29640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:48.578 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.578+0000 7f53a4e29640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:48.578 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.578+0000 7f53a4e29640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:26:48.579 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.578+0000 7f53a4e29640 1 -- 172.21.15.122:0/2392537511 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53a00b0d50 con 0x7f53a00b4d60 2024-03-25T18:26:48.583 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.582+0000 7f53a4e29640 1 -- 172.21.15.122:0/2392537511 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2223529095 0 0) 0x7f53a00af940 con 0x7f53a00b1730 2024-03-25T18:26:48.583 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.582+0000 7f53a4e29640 1 -- 172.21.15.122:0/2392537511 >> v1:172.21.15.204:6789/0 conn(0x7f53a00b4d60 legacy=0x7f53a013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.583 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.582+0000 7f53a4e29640 1 -- 172.21.15.122:0/2392537511 >> v1:172.21.15.122:6789/0 conn(0x7f53a014be90 legacy=0x7f53a014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.583 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.582+0000 7f53a4e29640 10 monclient: _finish_hunting 0 2024-03-25T18:26:48.583 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.582+0000 7f53a4e29640 1 monclient: found mon.noname-b 2024-03-25T18:26:48.583 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.582+0000 7f53a4e29640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:48.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.582+0000 7f53a4e29640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:26:48.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.582+0000 7f53a4e29640 1 -- 172.21.15.122:0/2392537511 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a00b1030 con 0x7f53a00b1730 2024-03-25T18:26:48.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.582+0000 7f53a4e29640 10 monclient: _finish_auth 0 2024-03-25T18:26:48.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.582+0000 7f53a4e29640 10 monclient: _check_auth_tickets 2024-03-25T18:26:48.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.582+0000 7f53a4e29640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:48.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.582+0000 7f53a4e29640 1 -- 172.21.15.122:0/2392537511 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5390001670 con 0x7f53a00b1730 2024-03-25T18:26:48.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.582+0000 7f53a4e29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:48.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.582+0000 7f53a4e29640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:48.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.582+0000 7f53a4e29640 10 monclient: dump: 2024-03-25T18:26:48.584 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:48.584 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:48.584 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:48.585 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:48.585 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:48.585 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:48.585 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:48.585 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:48.585 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:48.585 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:48.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.582+0000 7f53a808f640 5 monclient: authenticate success, global_id 4652 2024-03-25T18:26:48.585 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.582+0000 7f53a808f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:48.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.599+0000 7f53a4e29640 1 -- 172.21.15.122:0/2392537511 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53900017d0 con 0x7f53a00b1730 2024-03-25T18:26:48.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.600+0000 7f53a4e29640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:48.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.600+0000 7f53a4e29640 1 -- 172.21.15.122:0/2392537511 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5390001be0 con 0x7f53a00b1730 2024-03-25T18:26:48.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.600+0000 7f53a4e29640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:48.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.600+0000 7f53a4e29640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:48.600 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.600+0000 7f53a4e29640 10 monclient: dump: 2024-03-25T18:26:48.600 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:48.601 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:48.601 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:48.601 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:48.601 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:48.601 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:48.601 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:48.601 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:48.601 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:48.601 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:48.601 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.600+0000 7f53a808f640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:48.601 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.600+0000 7f53a808f640 4 set_mon_vals no callback set 2024-03-25T18:26:48.604 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.600+0000 7f53a808f640 10 monclient: shutdown 2024-03-25T18:26:48.604 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.600+0000 7f53a808f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:48.604 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.600+0000 7f53a808f640 1 -- 172.21.15.122:0/2392537511 >> v1:172.21.15.122:6790/0 conn(0x7f53a00b1730 legacy=0x7f53a00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.604 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.600+0000 7f53a808f640 1 -- 172.21.15.122:0/2392537511 shutdown_connections 2024-03-25T18:26:48.604 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.600+0000 7f53a808f640 1 -- 172.21.15.122:0/2392537511 >> 172.21.15.122:0/2392537511 conn(0x7f53a00617a0 msgr2=0x7f53a0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:48.604 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.601+0000 7f53a808f640 1 -- 172.21.15.122:0/2392537511 shutdown_connections 2024-03-25T18:26:48.604 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.601+0000 7f53a808f640 1 -- 172.21.15.122:0/2392537511 wait complete. 2024-03-25T18:26:48.604 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.601+0000 7f53a808f640 10 monclient: build_initial_monmap 2024-03-25T18:26:48.604 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.601+0000 7f53a808f640 10 monclient: monmap: 2024-03-25T18:26:48.604 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:26:48.604 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:48.604 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:26:48.604 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:26:48.604 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:48.605 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:26:48.605 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:26:48.605 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:26:48.605 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:26:48.605 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:48.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.601+0000 7f53a808f640 1 librados: starting msgr at 2024-03-25T18:26:48.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.602+0000 7f53a808f640 1 librados: starting objecter 2024-03-25T18:26:48.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.602+0000 7f53a808f640 1 Processor -- start 2024-03-25T18:26:48.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.602+0000 7f53a808f640 1 -- start start 2024-03-25T18:26:48.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.602+0000 7f53a808f640 1 librados: setting wanted keys 2024-03-25T18:26:48.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.602+0000 7f53a808f640 1 librados: calling monclient init 2024-03-25T18:26:48.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.602+0000 7f53a808f640 10 monclient: init 2024-03-25T18:26:48.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.602+0000 7f53a808f640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:48.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.602+0000 7f53a808f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:48.605 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.602+0000 7f53a808f640 10 monclient(hunting): picked mon.noname-b con 0x7f53a00b1730 addr v1:172.21.15.204:6789/0 2024-03-25T18:26:48.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.602+0000 7f53a808f640 10 monclient(hunting): picked mon.noname-a con 0x7f53a00b4d60 addr v1:172.21.15.122:6789/0 2024-03-25T18:26:48.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.602+0000 7f53a808f640 10 monclient(hunting): picked mon.noname-c con 0x7f53a014be90 addr v1:172.21.15.122:6790/0 2024-03-25T18:26:48.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.602+0000 7f53a808f640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a00a9610 con 0x7f53a00b4d60 2024-03-25T18:26:48.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.602+0000 7f53a808f640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a01433d0 con 0x7f53a00b1730 2024-03-25T18:26:48.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.602+0000 7f53a808f640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53a0143560 con 0x7f53a014be90 2024-03-25T18:26:48.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.602+0000 7f53a808f640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:48.606 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.602+0000 7f53a808f640 10 monclient(hunting): authenticate will time out at 7931.867697s 2024-03-25T18:26:48.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.619+0000 7f53a5e2b640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f53a00b1730 0x7f53a01d79a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:39708/0 (socket says 172.21.15.122:39708) 2024-03-25T18:26:48.619 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.619+0000 7f53a5e2b640 1 -- 172.21.15.122:0/2902640289 learned_addr learned my addr 172.21.15.122:0/2902640289 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:26:48.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.620+0000 7f53967fc640 10 client.?.objecter ms_handle_connect 0x7f53a00b1730 2024-03-25T18:26:48.620 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.620+0000 7f53967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:48.628 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.627+0000 7f53967fc640 10 client.?.objecter ms_handle_connect 0x7f53a014be90 2024-03-25T18:26:48.628 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.627+0000 7f53967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:48.632 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.631+0000 7f53967fc640 10 client.?.objecter ms_handle_connect 0x7f53a00b4d60 2024-03-25T18:26:48.632 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.631+0000 7f53967fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:48.632 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.632+0000 7f53967fc640 1 -- 172.21.15.122:0/2902640289 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3592303411 0 0) 0x7f53a01433d0 con 0x7f53a00b1730 2024-03-25T18:26:48.632 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.632+0000 7f53967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:48.633 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.632+0000 7f53967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:48.633 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.632+0000 7f53967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:48.633 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.632+0000 7f53967fc640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:26:48.633 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.632+0000 7f53967fc640 1 -- 172.21.15.122:0/2902640289 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5384004eb0 con 0x7f53a00b1730 2024-03-25T18:26:48.638 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.637+0000 7f53967fc640 1 -- 172.21.15.122:0/2902640289 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3359448400 0 0) 0x7f5384004eb0 con 0x7f53a00b1730 2024-03-25T18:26:48.638 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.637+0000 7f53967fc640 1 -- 172.21.15.122:0/2902640289 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53a01433d0 con 0x7f53a00b1730 2024-03-25T18:26:48.638 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.637+0000 7f53967fc640 1 -- 172.21.15.122:0/2902640289 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f539c001a10 con 0x7f53a00b1730 2024-03-25T18:26:48.638 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.637+0000 7f53967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:48.638 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.637+0000 7f53967fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:48.638 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.637+0000 7f53967fc640 10 monclient(hunting): dump: 2024-03-25T18:26:48.638 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:48.638 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:48.638 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:48.638 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:48.638 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:48.639 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:48.639 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:48.639 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:48.639 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:48.639 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:48.639 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.637+0000 7f53967fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:48.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.642+0000 7f53967fc640 1 -- 172.21.15.122:0/2902640289 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061406342 0 0) 0x7f53a00a9610 con 0x7f53a00b4d60 2024-03-25T18:26:48.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.642+0000 7f53967fc640 10 monclient(hunting): _negotiate 2024-03-25T18:26:48.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.642+0000 7f53967fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:48.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.642+0000 7f53967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:48.642 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.642+0000 7f53967fc640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:26:48.643 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.642+0000 7f53967fc640 1 -- 172.21.15.122:0/2902640289 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5384004eb0 con 0x7f53a00b4d60 2024-03-25T18:26:48.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53967fc640 1 -- 172.21.15.122:0/2902640289 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 56466789 0 0) 0x7f53a01433d0 con 0x7f53a00b1730 2024-03-25T18:26:48.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53967fc640 1 -- 172.21.15.122:0/2902640289 >> v1:172.21.15.122:6790/0 conn(0x7f53a014be90 legacy=0x7f53a01e2210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53967fc640 1 -- 172.21.15.122:0/2902640289 >> v1:172.21.15.122:6789/0 conn(0x7f53a00b4d60 legacy=0x7f53a01d80b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:48.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53967fc640 10 monclient: _finish_hunting 0 2024-03-25T18:26:48.645 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53967fc640 1 monclient: found mon.b 2024-03-25T18:26:48.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:48.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53967fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:26:48.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53967fc640 1 -- 172.21.15.122:0/2902640289 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53a01436f0 con 0x7f53a00b1730 2024-03-25T18:26:48.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53967fc640 10 monclient: _finish_auth 0 2024-03-25T18:26:48.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53967fc640 10 monclient: _check_auth_tickets 2024-03-25T18:26:48.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:48.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53a808f640 5 monclient: authenticate success, global_id 4687 2024-03-25T18:26:48.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53a808f640 10 monclient: _renew_subs 2024-03-25T18:26:48.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53a808f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:26:48.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53a808f640 1 -- 172.21.15.122:0/2902640289 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a01429a0 con 0x7f53a00b1730 2024-03-25T18:26:48.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53a808f640 10 client.4687.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:48.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53a808f640 10 monclient: _renew_subs 2024-03-25T18:26:48.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53a808f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:26:48.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53a808f640 1 -- 172.21.15.122:0/2902640289 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a0140560 con 0x7f53a00b1730 2024-03-25T18:26:48.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.644+0000 7f53a808f640 1 librados: init done 2024-03-25T18:26:48.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.645+0000 7f53a808f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:48.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.645+0000 7f53a808f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:48.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.645+0000 7f53a808f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:26:48.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.645+0000 7f53a808f640 1 -- 172.21.15.122:0/2902640289 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a0061d10 con 0x7f53a00b1730 2024-03-25T18:26:48.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.908+0000 7f53967fc640 1 -- 172.21.15.122:0/2902640289 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f539c001cb0 con 0x7f53a00b1730 2024-03-25T18:26:48.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.908+0000 7f53967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:48.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.908+0000 7f53967fc640 1 -- 172.21.15.122:0/2902640289 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f539c003a60 con 0x7f53a00b1730 2024-03-25T18:26:48.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.908+0000 7f53967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:48.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.908+0000 7f53967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:48.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.908+0000 7f53967fc640 10 monclient: dump: 2024-03-25T18:26:48.909 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:48.909 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:48.909 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:48.909 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:48.910 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:48.910 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:48.910 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:48.910 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:48.910 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:48.910 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:48.910 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.908+0000 7f5397fff640 4 set_mon_vals no callback set 2024-03-25T18:26:48.911 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.909+0000 7f53967fc640 1 -- 172.21.15.122:0/2902640289 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f539c003c80 con 0x7f53a00b1730 2024-03-25T18:26:48.911 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.910+0000 7f53a562a640 10 monclient: get_auth_request con 0x7f538406dd90 auth_method 0 2024-03-25T18:26:48.911 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.910+0000 7f53967fc640 10 client.4687.objecter ms_handle_connect 0x7f538406dd90 2024-03-25T18:26:48.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.916+0000 7f53967fc640 1 -- 172.21.15.122:0/2902640289 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1633957047 0 0) 0x7f539c054860 con 0x7f53a00b1730 2024-03-25T18:26:48.916 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.916+0000 7f53967fc640 10 client.4687.objecter ms_dispatch 0x7f53a0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:26:48.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.916+0000 7f53967fc640 3 client.4687.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:26:48.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.916+0000 7f53967fc640 3 client.4687.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:26:48.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.916+0000 7f53967fc640 20 client.4687.objecter dump_active .. 0 homeless 2024-03-25T18:26:48.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.917+0000 7f53967fc640 1 -- 172.21.15.122:0/2902640289 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f539c050370 con 0x7f53a00b1730 2024-03-25T18:26:48.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.917+0000 7f53967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:48.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:48.917+0000 7f53967fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:49.153 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.152+0000 7f53a808f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:49.153 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.152+0000 7f53a808f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:49.153 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.152+0000 7f53a808f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:26:49.153 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.152+0000 7f53a808f640 1 -- 172.21.15.122:0/2902640289 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53a00aaa20 con 0x7f53a00b1730 2024-03-25T18:26:49.214 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.213+0000 7f53967fc640 1 -- 172.21.15.122:0/2902640289 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1879996311) 0x7f539c0522a0 con 0x7f53a00b1730 2024-03-25T18:26:49.214 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.213+0000 7f53967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:49.214 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.213+0000 7f53967fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:49.214 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:26:49.215 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":11,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:26:32.649240+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:32.649240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:49.217 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.216+0000 7f53a808f640 10 librados: watch_flush enter 2024-03-25T18:26:49.217 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.216+0000 7f53a808f640 10 librados: watch_flush exit 2024-03-25T18:26:49.217 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.216+0000 7f53a808f640 20 client.4687.objecter shutdown clearing up homeless session... 2024-03-25T18:26:49.217 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.216+0000 7f53a808f640 10 client.4687.objecter successfully canceled tick 2024-03-25T18:26:49.217 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.216+0000 7f53a808f640 1 -- 172.21.15.122:0/2902640289 >> v1:172.21.15.204:6800/676228005 conn(0x7f538406dd90 legacy=0x7f5384070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:49.217 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.216+0000 7f53a808f640 10 monclient: shutdown 2024-03-25T18:26:49.217 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.216+0000 7f53a808f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:49.217 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.216+0000 7f53a808f640 1 -- 172.21.15.122:0/2902640289 >> v1:172.21.15.204:6789/0 conn(0x7f53a00b1730 legacy=0x7f53a01d79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:49.217 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.216+0000 7f53a808f640 1 -- 172.21.15.122:0/2902640289 shutdown_connections 2024-03-25T18:26:49.217 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.216+0000 7f53a808f640 1 -- 172.21.15.122:0/2902640289 >> 172.21.15.122:0/2902640289 conn(0x7f53a00617a0 msgr2=0x7f53a013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:49.217 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.216+0000 7f53a808f640 1 -- 172.21.15.122:0/2902640289 shutdown_connections 2024-03-25T18:26:49.217 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.216+0000 7f53a808f640 1 -- 172.21.15.122:0/2902640289 wait complete. 2024-03-25T18:26:49.217 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:49.216+0000 7f53a808f640 1 librados: shutdown 2024-03-25T18:26:49.228 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:56.035 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:26:56.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.148+0000 7f27863f4640 10 monclient: get_monmap_and_config 2024-03-25T18:26:56.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.148+0000 7f27863f4640 10 monclient: build_initial_monmap 2024-03-25T18:26:56.150 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.148+0000 7f27863f4640 10 monclient: monmap: 2024-03-25T18:26:56.150 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:26:56.150 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:56.150 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:26:56.149775+0000 2024-03-25T18:26:56.150 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:26:56.149775+0000 2024-03-25T18:26:56.150 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:56.150 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:56.151 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:26:56.151 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:26:56.151 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:26:56.151 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:56.152 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.149+0000 7f27863f4640 1 Processor -- start 2024-03-25T18:26:56.152 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.149+0000 7f27863f4640 1 -- start start 2024-03-25T18:26:56.152 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.149+0000 7f27863f4640 10 monclient: init 2024-03-25T18:26:56.152 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.150+0000 7f27863f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:56.152 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.150+0000 7f27863f4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:26:56.152 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.150+0000 7f27863f4640 10 monclient(hunting): picked mon.noname-c con 0x7f27800b4d60 addr v1:172.21.15.204:6789/0 2024-03-25T18:26:56.152 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.150+0000 7f27863f4640 10 monclient(hunting): picked mon.noname-a con 0x7f27800b1730 addr v1:172.21.15.122:6789/0 2024-03-25T18:26:56.152 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.150+0000 7f27863f4640 10 monclient(hunting): picked mon.noname-b con 0x7f278014be90 addr v1:172.21.15.122:6790/0 2024-03-25T18:26:56.152 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.150+0000 7f27863f4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27800af940 con 0x7f27800b1730 2024-03-25T18:26:56.152 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.150+0000 7f27863f4640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27800b0a70 con 0x7f27800b4d60 2024-03-25T18:26:56.153 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.150+0000 7f27863f4640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27800b0d50 con 0x7f278014be90 2024-03-25T18:26:56.153 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.150+0000 7f27863f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:56.153 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.150+0000 7f27863f4640 10 monclient(hunting): authenticate will time out at 7939.416122s 2024-03-25T18:26:56.155 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.154+0000 7f277f7fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f27800b1730 0x7f27800b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:46176/0 (socket says 172.21.15.122:46176) 2024-03-25T18:26:56.155 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.154+0000 7f277f7fe640 1 -- 172.21.15.122:0/3928378221 learned_addr learned my addr 172.21.15.122:0/3928378221 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:26:56.170 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.169+0000 7f277effd640 1 -- 172.21.15.122:0/3928378221 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2078409393 0 0) 0x7f27800af940 con 0x7f27800b1730 2024-03-25T18:26:56.170 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.169+0000 7f277effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:56.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.169+0000 7f277effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:56.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.169+0000 7f277effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:56.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.169+0000 7f277effd640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:26:56.171 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.169+0000 7f277effd640 1 -- 172.21.15.122:0/3928378221 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2768003520 con 0x7f27800b1730 2024-03-25T18:26:56.178 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.177+0000 7f277effd640 1 -- 172.21.15.122:0/3928378221 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3153155612 0 0) 0x7f27800b0d50 con 0x7f278014be90 2024-03-25T18:26:56.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.177+0000 7f277effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:56.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.177+0000 7f277effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:56.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.177+0000 7f277effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:56.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.177+0000 7f277effd640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:26:56.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.177+0000 7f277effd640 1 -- 172.21.15.122:0/3928378221 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27800af940 con 0x7f278014be90 2024-03-25T18:26:56.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.180+0000 7f277effd640 1 -- 172.21.15.122:0/3928378221 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1065329207 0 0) 0x7f27800b0a70 con 0x7f27800b4d60 2024-03-25T18:26:56.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.180+0000 7f277effd640 10 monclient(hunting): _negotiate 2024-03-25T18:26:56.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.180+0000 7f277effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:56.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.180+0000 7f277effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:56.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.180+0000 7f277effd640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:26:56.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.180+0000 7f277effd640 1 -- 172.21.15.122:0/3928378221 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27800b0d50 con 0x7f27800b4d60 2024-03-25T18:26:56.198 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.196+0000 7f277effd640 1 -- 172.21.15.122:0/3928378221 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1499003641 0 0) 0x7f27800af940 con 0x7f278014be90 2024-03-25T18:26:56.198 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.196+0000 7f277effd640 1 -- 172.21.15.122:0/3928378221 >> v1:172.21.15.204:6789/0 conn(0x7f27800b4d60 legacy=0x7f278013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:56.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.196+0000 7f277effd640 1 -- 172.21.15.122:0/3928378221 >> v1:172.21.15.122:6789/0 conn(0x7f27800b1730 legacy=0x7f27800b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:56.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.196+0000 7f277effd640 10 monclient: _finish_hunting 0 2024-03-25T18:26:56.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.196+0000 7f277effd640 1 monclient: found mon.noname-b 2024-03-25T18:26:56.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.196+0000 7f277effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:56.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.196+0000 7f277effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:26:56.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.196+0000 7f277effd640 1 -- 172.21.15.122:0/3928378221 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27800b1030 con 0x7f278014be90 2024-03-25T18:26:56.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.196+0000 7f277effd640 10 monclient: _finish_auth 0 2024-03-25T18:26:56.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.196+0000 7f277effd640 10 monclient: _check_auth_tickets 2024-03-25T18:26:56.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.196+0000 7f277effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:56.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.196+0000 7f277effd640 1 -- 172.21.15.122:0/3928378221 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f27780019f0 con 0x7f278014be90 2024-03-25T18:26:56.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.196+0000 7f277effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:56.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.197+0000 7f277effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:26:56.199 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.197+0000 7f277effd640 10 monclient: dump: 2024-03-25T18:26:56.199 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:56.200 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:56.200 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:56.200 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:56.200 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:56.200 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:56.200 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:56.200 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:56.200 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:56.200 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:56.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.197+0000 7f27863f4640 5 monclient: authenticate success, global_id 4658 2024-03-25T18:26:56.200 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.197+0000 7f27863f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:26:56.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.224+0000 7f277effd640 1 -- 172.21.15.122:0/3928378221 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2778001b50 con 0x7f278014be90 2024-03-25T18:26:56.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.224+0000 7f277effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:56.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.224+0000 7f277effd640 1 -- 172.21.15.122:0/3928378221 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f2778001f60 con 0x7f278014be90 2024-03-25T18:26:56.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.224+0000 7f277effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:56.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.224+0000 7f277effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:56.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.224+0000 7f277effd640 10 monclient: dump: 2024-03-25T18:26:56.226 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:56.226 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:56.226 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:56.226 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:56.226 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:56.226 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:56.226 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:56.226 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:56.226 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:56.226 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:56.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.224+0000 7f27863f4640 10 monclient: get_monmap_and_config success 2024-03-25T18:26:56.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.224+0000 7f27863f4640 4 set_mon_vals no callback set 2024-03-25T18:26:56.229 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.225+0000 7f27863f4640 10 monclient: shutdown 2024-03-25T18:26:56.229 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.225+0000 7f27863f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:56.229 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.225+0000 7f27863f4640 1 -- 172.21.15.122:0/3928378221 >> v1:172.21.15.122:6790/0 conn(0x7f278014be90 legacy=0x7f278014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:56.229 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.225+0000 7f27863f4640 1 -- 172.21.15.122:0/3928378221 shutdown_connections 2024-03-25T18:26:56.229 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.225+0000 7f27863f4640 1 -- 172.21.15.122:0/3928378221 >> 172.21.15.122:0/3928378221 conn(0x7f27800617a0 msgr2=0x7f2780061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:56.229 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.225+0000 7f27863f4640 1 -- 172.21.15.122:0/3928378221 shutdown_connections 2024-03-25T18:26:56.229 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.225+0000 7f27863f4640 1 -- 172.21.15.122:0/3928378221 wait complete. 2024-03-25T18:26:56.229 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.226+0000 7f27863f4640 10 monclient: build_initial_monmap 2024-03-25T18:26:56.229 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.226+0000 7f27863f4640 10 monclient: monmap: 2024-03-25T18:26:56.229 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:26:56.229 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:26:56.230 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:26:56.230 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:26:56.230 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:26:56.230 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:26:56.230 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:26:56.230 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:26:56.230 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:26:56.230 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:56.230 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.226+0000 7f27863f4640 1 librados: starting msgr at 2024-03-25T18:26:56.230 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.226+0000 7f27863f4640 1 librados: starting objecter 2024-03-25T18:26:56.230 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.226+0000 7f27863f4640 1 Processor -- start 2024-03-25T18:26:56.230 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.226+0000 7f27863f4640 1 -- start start 2024-03-25T18:26:56.230 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.226+0000 7f27863f4640 1 librados: setting wanted keys 2024-03-25T18:26:56.230 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.226+0000 7f27863f4640 1 librados: calling monclient init 2024-03-25T18:26:56.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.226+0000 7f27863f4640 10 monclient: init 2024-03-25T18:26:56.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.227+0000 7f27863f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:26:56.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.227+0000 7f27863f4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:26:56.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.227+0000 7f27863f4640 10 monclient(hunting): picked mon.noname-c con 0x7f27800b1730 addr v1:172.21.15.122:6790/0 2024-03-25T18:26:56.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.227+0000 7f27863f4640 10 monclient(hunting): picked mon.noname-a con 0x7f27800b4d60 addr v1:172.21.15.122:6789/0 2024-03-25T18:26:56.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.227+0000 7f27863f4640 10 monclient(hunting): picked mon.noname-b con 0x7f278014be90 addr v1:172.21.15.204:6789/0 2024-03-25T18:26:56.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.227+0000 7f27863f4640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27801d77c0 con 0x7f27800b4d60 2024-03-25T18:26:56.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.227+0000 7f27863f4640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2780141f80 con 0x7f278014be90 2024-03-25T18:26:56.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.227+0000 7f27863f4640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2780142260 con 0x7f27800b1730 2024-03-25T18:26:56.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.227+0000 7f27863f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:26:56.231 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.227+0000 7f27863f4640 10 monclient(hunting): authenticate will time out at 7939.493608s 2024-03-25T18:26:56.239 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.237+0000 7f277ffff640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f27800b1730 0x7f27801d7c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:44802/0 (socket says 172.21.15.122:44802) 2024-03-25T18:26:56.239 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.237+0000 7f277ffff640 1 -- 172.21.15.122:0/4261959602 learned_addr learned my addr 172.21.15.122:0/4261959602 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:26:56.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.239+0000 7f277cff9640 10 client.?.objecter ms_handle_connect 0x7f27800b1730 2024-03-25T18:26:56.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.239+0000 7f277cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:56.246 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.245+0000 7f277cff9640 10 client.?.objecter ms_handle_connect 0x7f27800b4d60 2024-03-25T18:26:56.246 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.245+0000 7f277cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:56.247 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.246+0000 7f277cff9640 10 client.?.objecter ms_handle_connect 0x7f278014be90 2024-03-25T18:26:56.247 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.246+0000 7f277cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:26:56.254 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.253+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1155937459 0 0) 0x7f2780142260 con 0x7f27800b1730 2024-03-25T18:26:56.255 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.253+0000 7f277cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:56.255 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.253+0000 7f277cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:56.255 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.253+0000 7f277cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:56.255 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.253+0000 7f277cff9640 10 monclient(hunting): my global_id is 4661 2024-03-25T18:26:56.255 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.253+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f276c004eb0 con 0x7f27800b1730 2024-03-25T18:26:56.258 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.257+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4063807066 0 0) 0x7f2780141f80 con 0x7f278014be90 2024-03-25T18:26:56.258 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.257+0000 7f277cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:56.258 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.257+0000 7f277cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:56.258 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.257+0000 7f277cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:56.258 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.257+0000 7f277cff9640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:26:56.258 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.257+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2780142260 con 0x7f278014be90 2024-03-25T18:26:56.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.259+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024653184 0 0) 0x7f27801d77c0 con 0x7f27800b4d60 2024-03-25T18:26:56.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.259+0000 7f277cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:26:56.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.259+0000 7f277cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:26:56.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.259+0000 7f277cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:26:56.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.259+0000 7f277cff9640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:26:56.261 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.259+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2780141f80 con 0x7f27800b4d60 2024-03-25T18:26:56.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.266+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1814236549 0 0) 0x7f276c004eb0 con 0x7f27800b1730 2024-03-25T18:26:56.268 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.266+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27801d77c0 con 0x7f27800b1730 2024-03-25T18:26:56.268 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.266+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f2770001a10 con 0x7f27800b1730 2024-03-25T18:26:56.268 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.266+0000 7f277cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:56.268 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.266+0000 7f277cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:26:56.268 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.266+0000 7f277cff9640 10 monclient(hunting): dump: 2024-03-25T18:26:56.268 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:56.268 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:56.268 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:56.268 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:56.268 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:56.268 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:56.268 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:56.268 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:56.268 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:56.269 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:56.269 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.266+0000 7f277cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:26:56.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.271+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1736563576 0 0) 0x7f2780142260 con 0x7f278014be90 2024-03-25T18:26:56.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.271+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f276c004eb0 con 0x7f278014be90 2024-03-25T18:26:56.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.271+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f2778001e60 con 0x7f278014be90 2024-03-25T18:26:56.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.271+0000 7f277cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:56.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.271+0000 7f277cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:26:56.272 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.271+0000 7f277cff9640 10 monclient(hunting): dump: 2024-03-25T18:26:56.272 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:56.272 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:56.273 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:56.273 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:56.273 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:56.273 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:56.273 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:56.273 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:56.273 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:56.273 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:56.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3169053971 0 0) 0x7f27801d77c0 con 0x7f27800b1730 2024-03-25T18:26:56.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 >> v1:172.21.15.204:6789/0 conn(0x7f278014be90 legacy=0x7f27801e3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:56.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 >> v1:172.21.15.122:6789/0 conn(0x7f27800b4d60 legacy=0x7f2780142c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:56.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f277cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:26:56.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f277cff9640 1 monclient: found mon.c 2024-03-25T18:26:56.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f277cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:26:56.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f277cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:56.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27801426f0 con 0x7f27800b1730 2024-03-25T18:26:56.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f277cff9640 10 monclient: _finish_auth 0 2024-03-25T18:26:56.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f277cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:26:56.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f277cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:26:56.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f27863f4640 5 monclient: authenticate success, global_id 4661 2024-03-25T18:26:56.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f27863f4640 10 monclient: _renew_subs 2024-03-25T18:26:56.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f27863f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:56.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f27863f4640 1 -- 172.21.15.122:0/4261959602 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27801d8360 con 0x7f27800b1730 2024-03-25T18:26:56.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f27863f4640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:26:56.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f27863f4640 10 monclient: _renew_subs 2024-03-25T18:26:56.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f27863f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:56.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f27863f4640 1 -- 172.21.15.122:0/4261959602 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2780143540 con 0x7f27800b1730 2024-03-25T18:26:56.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.277+0000 7f27863f4640 1 librados: init done 2024-03-25T18:26:56.281 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.278+0000 7f27863f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:56.281 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.278+0000 7f27863f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:56.281 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.278+0000 7f27863f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:56.281 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.278+0000 7f27863f4640 1 -- 172.21.15.122:0/4261959602 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2780142880 con 0x7f27800b1730 2024-03-25T18:26:56.577 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.575+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2770001c70 con 0x7f27800b1730 2024-03-25T18:26:56.577 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.575+0000 7f277cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:26:56.577 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.575+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f27700038c0 con 0x7f27800b1730 2024-03-25T18:26:56.577 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.575+0000 7f277cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:26:56.577 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.575+0000 7f277cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:26:56.577 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.575+0000 7f277cff9640 10 monclient: dump: 2024-03-25T18:26:56.577 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:26:56.577 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:26:56.578 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:56.578 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:26:56.578 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:26:56.578 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:26:56.578 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:26:56.578 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:26:56.578 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:26:56.578 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:26:56.578 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.575+0000 7f277e7fc640 4 set_mon_vals no callback set 2024-03-25T18:26:56.579 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.576+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f2770003aa0 con 0x7f27800b1730 2024-03-25T18:26:56.579 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.577+0000 7f277f7fe640 10 monclient: get_auth_request con 0x7f276c06df70 auth_method 0 2024-03-25T18:26:56.579 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.577+0000 7f277cff9640 10 client.4661.objecter ms_handle_connect 0x7f276c06df70 2024-03-25T18:26:56.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.582+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1633957047 0 0) 0x7f2770004460 con 0x7f27800b1730 2024-03-25T18:26:56.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.582+0000 7f277cff9640 10 client.4661.objecter ms_dispatch 0x7f2780000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:26:56.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.582+0000 7f277cff9640 3 client.4661.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:26:56.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.582+0000 7f277cff9640 3 client.4661.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:26:56.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.582+0000 7f277cff9640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-25T18:26:56.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.582+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27700500f0 con 0x7f27800b1730 2024-03-25T18:26:56.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.582+0000 7f277cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:26:56.584 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.582+0000 7f277cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:26:56.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.811+0000 7f27863f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:56.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.811+0000 7f27863f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:56.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.811+0000 7f27863f4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:26:56.813 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.811+0000 7f27863f4640 1 -- 172.21.15.122:0/4261959602 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27801d86b0 con 0x7f27800b1730 2024-03-25T18:26:56.897 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.895+0000 7f277cff9640 1 -- 172.21.15.122:0/4261959602 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1879996311) 0x7f27700521f0 con 0x7f27800b1730 2024-03-25T18:26:56.897 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.896+0000 7f277cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:26:56.897 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.896+0000 7f277cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:26:56.897 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:26:56.898 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":11,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:26:32.649240+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:32.649240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:26:56.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.898+0000 7f27863f4640 10 librados: watch_flush enter 2024-03-25T18:26:56.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.898+0000 7f27863f4640 10 librados: watch_flush exit 2024-03-25T18:26:56.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.898+0000 7f27863f4640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-25T18:26:56.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.898+0000 7f27863f4640 10 client.4661.objecter successfully canceled tick 2024-03-25T18:26:56.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.898+0000 7f27863f4640 1 -- 172.21.15.122:0/4261959602 >> v1:172.21.15.204:6800/676228005 conn(0x7f276c06df70 legacy=0x7f276c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:56.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.898+0000 7f27863f4640 10 monclient: shutdown 2024-03-25T18:26:56.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.898+0000 7f27863f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:26:56.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.898+0000 7f27863f4640 1 -- 172.21.15.122:0/4261959602 >> v1:172.21.15.122:6790/0 conn(0x7f27800b1730 legacy=0x7f27801d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:26:56.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.898+0000 7f27863f4640 1 -- 172.21.15.122:0/4261959602 shutdown_connections 2024-03-25T18:26:56.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.898+0000 7f27863f4640 1 -- 172.21.15.122:0/4261959602 >> 172.21.15.122:0/4261959602 conn(0x7f27800617a0 msgr2=0x7f27800b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:26:56.900 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.898+0000 7f27863f4640 1 -- 172.21.15.122:0/4261959602 shutdown_connections 2024-03-25T18:26:56.901 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.898+0000 7f27863f4640 1 -- 172.21.15.122:0/4261959602 wait complete. 2024-03-25T18:26:56.901 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:26:56.898+0000 7f27863f4640 1 librados: shutdown 2024-03-25T18:26:56.911 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:26:57.833 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:10:51.575 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:57.836 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:10:51.578 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:57.874 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:10:51.616 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:57.901 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:10:51.647 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:57.905 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:10:51.650 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:57.910 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:10:51.653 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:57.914 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:10:51.656 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:57.933 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:10:51.678 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:57.941 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:10:51.683 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:03.049 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:10:56.791 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:03.057 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:10:56.799 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:03.066 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:10:56.809 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:03.068 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:10:56.813 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:03.069 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:10:56.811 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:03.074 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:10:56.816 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:03.079 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:10:56.821 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:03.089 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:10:56.835 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:03.092 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:10:56.834 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:03.718 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:03.829 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.828+0000 7f23b80dd640 10 monclient: get_monmap_and_config 2024-03-25T18:27:03.830 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.828+0000 7f23b80dd640 10 monclient: build_initial_monmap 2024-03-25T18:27:03.830 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.828+0000 7f23b80dd640 10 monclient: monmap: 2024-03-25T18:27:03.830 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:27:03.830 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:03.830 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:27:03.829636+0000 2024-03-25T18:27:03.830 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:27:03.829636+0000 2024-03-25T18:27:03.830 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:03.830 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:03.830 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:27:03.830 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:27:03.830 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:27:03.830 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:03.831 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.829+0000 7f23b80dd640 1 Processor -- start 2024-03-25T18:27:03.831 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.829+0000 7f23b80dd640 1 -- start start 2024-03-25T18:27:03.831 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.829+0000 7f23b80dd640 10 monclient: init 2024-03-25T18:27:03.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.830+0000 7f23b80dd640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:03.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.830+0000 7f23b80dd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:03.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.830+0000 7f23b80dd640 10 monclient(hunting): picked mon.noname-c con 0x7f23b00a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:27:03.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.830+0000 7f23b80dd640 10 monclient(hunting): picked mon.noname-b con 0x7f23b013f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:27:03.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.830+0000 7f23b80dd640 10 monclient(hunting): picked mon.noname-a con 0x7f23b01430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:27:03.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.830+0000 7f23b80dd640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b0142520 con 0x7f23b01430c0 2024-03-25T18:27:03.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.830+0000 7f23b80dd640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b00b28a0 con 0x7f23b00a98f0 2024-03-25T18:27:03.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.830+0000 7f23b80dd640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b00b2a30 con 0x7f23b013f710 2024-03-25T18:27:03.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.830+0000 7f23b80dd640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:03.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.830+0000 7f23b80dd640 10 monclient(hunting): authenticate will time out at 7947.096412s 2024-03-25T18:27:03.835 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.833+0000 7f23b5678640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f23b013f710 0x7f23b01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:45142/0 (socket says 172.21.15.122:45142) 2024-03-25T18:27:03.835 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.833+0000 7f23b5678640 1 -- 172.21.15.122:0/1826655051 learned_addr learned my addr 172.21.15.122:0/1826655051 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:27:03.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.845+0000 7f23b4e77640 1 -- 172.21.15.122:0/1826655051 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 883380813 0 0) 0x7f23b00b2a30 con 0x7f23b013f710 2024-03-25T18:27:03.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.845+0000 7f23b4e77640 10 monclient(hunting): _negotiate 2024-03-25T18:27:03.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.845+0000 7f23b4e77640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:03.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.845+0000 7f23b4e77640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:03.846 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.845+0000 7f23b4e77640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:27:03.847 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.845+0000 7f23b4e77640 1 -- 172.21.15.122:0/1826655051 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f239c003520 con 0x7f23b013f710 2024-03-25T18:27:03.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.862+0000 7f23b4e77640 1 -- 172.21.15.122:0/1826655051 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2054109463 0 0) 0x7f239c003520 con 0x7f23b013f710 2024-03-25T18:27:03.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.862+0000 7f23b4e77640 1 -- 172.21.15.122:0/1826655051 >> v1:172.21.15.204:6789/0 conn(0x7f23b00a98f0 legacy=0x7f23b00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:03.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.862+0000 7f23b4e77640 1 -- 172.21.15.122:0/1826655051 >> v1:172.21.15.122:6789/0 conn(0x7f23b01430c0 legacy=0x7f23b014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:03.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.862+0000 7f23b4e77640 10 monclient: _finish_hunting 0 2024-03-25T18:27:03.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.862+0000 7f23b4e77640 1 monclient: found mon.noname-b 2024-03-25T18:27:03.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.862+0000 7f23b4e77640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:03.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.862+0000 7f23b4e77640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:27:03.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.862+0000 7f23b4e77640 1 -- 172.21.15.122:0/1826655051 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b00b44c0 con 0x7f23b013f710 2024-03-25T18:27:03.864 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.862+0000 7f23b4e77640 10 monclient: _finish_auth 0 2024-03-25T18:27:03.865 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.862+0000 7f23b4e77640 10 monclient: _check_auth_tickets 2024-03-25T18:27:03.865 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.862+0000 7f23b4e77640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:03.865 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.862+0000 7f23b4e77640 1 -- 172.21.15.122:0/1826655051 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f23a0001690 con 0x7f23b013f710 2024-03-25T18:27:03.865 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.862+0000 7f23b4e77640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:03.865 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.862+0000 7f23b4e77640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:03.865 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.862+0000 7f23b4e77640 10 monclient: dump: 2024-03-25T18:27:03.865 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:03.865 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:03.865 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:03.865 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:03.865 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:03.865 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:03.865 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:03.865 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:03.865 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:03.866 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:03.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.862+0000 7f23b80dd640 5 monclient: authenticate success, global_id 4664 2024-03-25T18:27:03.866 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.862+0000 7f23b80dd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:03.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.871+0000 7f23b4e77640 1 -- 172.21.15.122:0/1826655051 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23a00017f0 con 0x7f23b013f710 2024-03-25T18:27:03.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.871+0000 7f23b4e77640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:03.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.871+0000 7f23b4e77640 1 -- 172.21.15.122:0/1826655051 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f23a0001c40 con 0x7f23b013f710 2024-03-25T18:27:03.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.871+0000 7f23b80dd640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:03.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.871+0000 7f23b80dd640 4 set_mon_vals no callback set 2024-03-25T18:27:03.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.871+0000 7f23b4e77640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:03.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.871+0000 7f23b4e77640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:03.873 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.871+0000 7f23b4e77640 10 monclient: dump: 2024-03-25T18:27:03.873 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:03.873 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:03.873 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:03.873 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:03.873 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:03.874 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:03.874 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:03.874 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:03.874 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:03.874 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:03.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.872+0000 7f23b80dd640 10 monclient: shutdown 2024-03-25T18:27:03.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.872+0000 7f23b80dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:03.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.872+0000 7f23b80dd640 1 -- 172.21.15.122:0/1826655051 >> v1:172.21.15.122:6790/0 conn(0x7f23b013f710 legacy=0x7f23b01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:03.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.872+0000 7f23b80dd640 1 -- 172.21.15.122:0/1826655051 shutdown_connections 2024-03-25T18:27:03.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.872+0000 7f23b80dd640 1 -- 172.21.15.122:0/1826655051 >> 172.21.15.122:0/1826655051 conn(0x7f23b00b7ea0 msgr2=0x7f23b00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:03.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.872+0000 7f23b80dd640 1 -- 172.21.15.122:0/1826655051 shutdown_connections 2024-03-25T18:27:03.876 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.872+0000 7f23b80dd640 1 -- 172.21.15.122:0/1826655051 wait complete. 2024-03-25T18:27:03.877 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.873+0000 7f23b80dd640 10 monclient: build_initial_monmap 2024-03-25T18:27:03.877 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.873+0000 7f23b80dd640 10 monclient: monmap: 2024-03-25T18:27:03.877 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:27:03.877 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:03.877 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:27:03.877 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:27:03.877 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:03.877 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:27:03.877 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:27:03.877 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:27:03.877 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:27:03.877 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:03.877 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.873+0000 7f23b80dd640 1 librados: starting msgr at 2024-03-25T18:27:03.877 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.873+0000 7f23b80dd640 1 librados: starting objecter 2024-03-25T18:27:03.877 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.873+0000 7f23b80dd640 1 Processor -- start 2024-03-25T18:27:03.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.873+0000 7f23b80dd640 1 -- start start 2024-03-25T18:27:03.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.873+0000 7f23b80dd640 1 librados: setting wanted keys 2024-03-25T18:27:03.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.873+0000 7f23b80dd640 1 librados: calling monclient init 2024-03-25T18:27:03.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.873+0000 7f23b80dd640 10 monclient: init 2024-03-25T18:27:03.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.874+0000 7f23b80dd640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:03.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.874+0000 7f23b80dd640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:03.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.874+0000 7f23b80dd640 10 monclient(hunting): picked mon.noname-b con 0x7f23b00a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:27:03.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.874+0000 7f23b80dd640 10 monclient(hunting): picked mon.noname-c con 0x7f23b013f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:27:03.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.874+0000 7f23b80dd640 10 monclient(hunting): picked mon.noname-a con 0x7f23b01430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:27:03.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.874+0000 7f23b80dd640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b0139a40 con 0x7f23b01430c0 2024-03-25T18:27:03.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.874+0000 7f23b80dd640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b0133f20 con 0x7f23b00a98f0 2024-03-25T18:27:03.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.874+0000 7f23b80dd640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23b00ab950 con 0x7f23b013f710 2024-03-25T18:27:03.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.874+0000 7f23b80dd640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:03.878 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.874+0000 7f23b80dd640 10 monclient(hunting): authenticate will time out at 7947.140669s 2024-03-25T18:27:03.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.891+0000 7f23b5678640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f23b013f710 0x7f23b0133100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:45188/0 (socket says 172.21.15.122:45188) 2024-03-25T18:27:03.892 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.891+0000 7f23b5678640 1 -- 172.21.15.122:0/1207785372 learned_addr learned my addr 172.21.15.122:0/1207785372 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:27:03.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.892+0000 7f23a67fc640 10 client.?.objecter ms_handle_connect 0x7f23b013f710 2024-03-25T18:27:03.893 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.892+0000 7f23a67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:03.903 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.902+0000 7f23a67fc640 10 client.?.objecter ms_handle_connect 0x7f23b00a98f0 2024-03-25T18:27:03.903 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.902+0000 7f23a67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:03.909 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.908+0000 7f23a67fc640 10 client.?.objecter ms_handle_connect 0x7f23b01430c0 2024-03-25T18:27:03.910 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.908+0000 7f23a67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:03.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.916+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109140885 0 0) 0x7f23b00ab950 con 0x7f23b013f710 2024-03-25T18:27:03.917 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.916+0000 7f23a67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:03.918 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.916+0000 7f23a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:03.918 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.916+0000 7f23a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:03.918 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.916+0000 7f23a67fc640 10 monclient(hunting): my global_id is 4667 2024-03-25T18:27:03.918 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.916+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2394004eb0 con 0x7f23b013f710 2024-03-25T18:27:03.920 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.919+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2032796297 0 0) 0x7f23b0139a40 con 0x7f23b01430c0 2024-03-25T18:27:03.920 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.919+0000 7f23a67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:03.921 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.919+0000 7f23a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:03.921 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.919+0000 7f23a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:03.921 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.919+0000 7f23a67fc640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:27:03.921 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.919+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b00ab950 con 0x7f23b01430c0 2024-03-25T18:27:03.924 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.923+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1911109575 0 0) 0x7f23b0133f20 con 0x7f23b00a98f0 2024-03-25T18:27:03.924 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.923+0000 7f23a67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:03.924 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.923+0000 7f23a67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:03.924 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.923+0000 7f23a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:03.924 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.923+0000 7f23a67fc640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:27:03.925 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.923+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23b0139a40 con 0x7f23b00a98f0 2024-03-25T18:27:03.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.932+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 749305234 0 0) 0x7f2394004eb0 con 0x7f23b013f710 2024-03-25T18:27:03.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.933+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f23b0133f20 con 0x7f23b013f710 2024-03-25T18:27:03.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.933+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f23a0001b00 con 0x7f23b013f710 2024-03-25T18:27:03.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.933+0000 7f23a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:03.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.933+0000 7f23a67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:03.934 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.933+0000 7f23a67fc640 10 monclient(hunting): dump: 2024-03-25T18:27:03.934 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:03.934 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:03.935 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:03.935 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:03.935 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:03.935 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:03.935 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:03.935 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:03.935 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:03.935 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:03.935 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.933+0000 7f23a67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:03.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.942+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1140523491 0 0) 0x7f23b0139a40 con 0x7f23b00a98f0 2024-03-25T18:27:03.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.942+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2394004eb0 con 0x7f23b00a98f0 2024-03-25T18:27:03.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.942+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f23ac001830 con 0x7f23b00a98f0 2024-03-25T18:27:03.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.942+0000 7f23a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:03.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.942+0000 7f23a67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:03.944 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.942+0000 7f23a67fc640 10 monclient(hunting): dump: 2024-03-25T18:27:03.944 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:03.944 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:03.944 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:03.945 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:03.945 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:03.945 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:03.945 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:03.945 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:03.945 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:03.945 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:03.958 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4125560532 0 0) 0x7f23b0133f20 con 0x7f23b013f710 2024-03-25T18:27:03.958 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 >> v1:172.21.15.204:6789/0 conn(0x7f23b00a98f0 legacy=0x7f23b0138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:03.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 >> v1:172.21.15.122:6789/0 conn(0x7f23b01430c0 legacy=0x7f23b0133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:03.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23a67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:03.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23a67fc640 1 monclient: found mon.c 2024-03-25T18:27:03.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:03.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23a67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:27:03.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23b00abc30 con 0x7f23b013f710 2024-03-25T18:27:03.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23a67fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:03.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23a67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:03.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:03.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23b80dd640 5 monclient: authenticate success, global_id 4667 2024-03-25T18:27:03.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23b80dd640 10 monclient: _renew_subs 2024-03-25T18:27:03.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23b80dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:27:03.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23b80dd640 1 -- 172.21.15.122:0/1207785372 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23b00abf10 con 0x7f23b013f710 2024-03-25T18:27:03.959 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23b80dd640 10 client.4667.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:03.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23b80dd640 10 monclient: _renew_subs 2024-03-25T18:27:03.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23b80dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:27:03.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23b80dd640 1 -- 172.21.15.122:0/1207785372 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f23b0139370 con 0x7f23b013f710 2024-03-25T18:27:03.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.957+0000 7f23b80dd640 1 librados: init done 2024-03-25T18:27:03.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.958+0000 7f23b80dd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:03.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.958+0000 7f23b80dd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:03.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.958+0000 7f23b80dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:27:03.960 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:03.958+0000 7f23b80dd640 1 -- 172.21.15.122:0/1207785372 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23b00627b0 con 0x7f23b013f710 2024-03-25T18:27:04.256 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.254+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f23a0001d20 con 0x7f23b013f710 2024-03-25T18:27:04.256 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.254+0000 7f23a67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:04.256 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.254+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f23a0004d90 con 0x7f23b013f710 2024-03-25T18:27:04.256 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.254+0000 7f23a67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:04.256 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.254+0000 7f23a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:04.256 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.254+0000 7f23a67fc640 10 monclient: dump: 2024-03-25T18:27:04.256 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:04.256 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:04.256 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:04.256 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:04.256 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:04.256 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:04.257 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:04.257 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:04.257 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:04.257 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:04.257 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.254+0000 7f23a7fff640 4 set_mon_vals no callback set 2024-03-25T18:27:04.257 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.255+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f23a001ad60 con 0x7f23b013f710 2024-03-25T18:27:04.257 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.256+0000 7f23b5e79640 10 monclient: get_auth_request con 0x7f239406ded0 auth_method 0 2024-03-25T18:27:04.258 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.256+0000 7f23a67fc640 10 client.4667.objecter ms_handle_connect 0x7f239406ded0 2024-03-25T18:27:04.265 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.264+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1633957047 0 0) 0x7f23a0054900 con 0x7f23b013f710 2024-03-25T18:27:04.265 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.264+0000 7f23a67fc640 10 client.4667.objecter ms_dispatch 0x7f23b0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:27:04.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.264+0000 7f23a67fc640 3 client.4667.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:27:04.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.264+0000 7f23a67fc640 3 client.4667.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:27:04.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.264+0000 7f23a67fc640 20 client.4667.objecter dump_active .. 0 homeless 2024-03-25T18:27:04.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.264+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f23a0050240 con 0x7f23b013f710 2024-03-25T18:27:04.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.264+0000 7f23a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:04.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.264+0000 7f23a67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:04.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.518+0000 7f23b80dd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:04.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.518+0000 7f23b80dd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:04.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.518+0000 7f23b80dd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:27:04.520 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.518+0000 7f23b80dd640 1 -- 172.21.15.122:0/1207785372 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23b00b7180 con 0x7f23b013f710 2024-03-25T18:27:04.543 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.542+0000 7f23a67fc640 1 -- 172.21.15.122:0/1207785372 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1879996311) 0x7f23a0001150 con 0x7f23b013f710 2024-03-25T18:27:04.544 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.542+0000 7f23a67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:04.544 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.542+0000 7f23a67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:04.544 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:27:04.544 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":11,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:26:32.649240+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:32.649240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:04.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.544+0000 7f23b80dd640 10 librados: watch_flush enter 2024-03-25T18:27:04.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.544+0000 7f23b80dd640 10 librados: watch_flush exit 2024-03-25T18:27:04.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.544+0000 7f23b80dd640 20 client.4667.objecter shutdown clearing up homeless session... 2024-03-25T18:27:04.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.544+0000 7f23b80dd640 10 client.4667.objecter successfully canceled tick 2024-03-25T18:27:04.546 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.544+0000 7f23b80dd640 1 -- 172.21.15.122:0/1207785372 >> v1:172.21.15.204:6800/676228005 conn(0x7f239406ded0 legacy=0x7f2394070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:04.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.544+0000 7f23b80dd640 10 monclient: shutdown 2024-03-25T18:27:04.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.544+0000 7f23b80dd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:04.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.544+0000 7f23b80dd640 1 -- 172.21.15.122:0/1207785372 >> v1:172.21.15.122:6790/0 conn(0x7f23b013f710 legacy=0x7f23b0133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:04.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.544+0000 7f23b80dd640 1 -- 172.21.15.122:0/1207785372 shutdown_connections 2024-03-25T18:27:04.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.544+0000 7f23b80dd640 1 -- 172.21.15.122:0/1207785372 >> 172.21.15.122:0/1207785372 conn(0x7f23b00b7ea0 msgr2=0x7f23b0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:04.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.545+0000 7f23b80dd640 1 -- 172.21.15.122:0/1207785372 shutdown_connections 2024-03-25T18:27:04.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.545+0000 7f23b80dd640 1 -- 172.21.15.122:0/1207785372 wait complete. 2024-03-25T18:27:04.547 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:04.545+0000 7f23b80dd640 1 librados: shutdown 2024-03-25T18:27:04.558 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:11.365 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:11.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.477+0000 7fe062e25640 10 monclient: get_monmap_and_config 2024-03-25T18:27:11.478 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.477+0000 7fe062e25640 10 monclient: build_initial_monmap 2024-03-25T18:27:11.479 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.477+0000 7fe062e25640 10 monclient: monmap: 2024-03-25T18:27:11.479 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:27:11.479 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:11.479 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:27:11.478451+0000 2024-03-25T18:27:11.479 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:27:11.478451+0000 2024-03-25T18:27:11.479 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:11.479 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:11.479 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:27:11.479 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:27:11.479 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:27:11.479 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:11.480 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.478+0000 7fe062e25640 1 Processor -- start 2024-03-25T18:27:11.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.478+0000 7fe062e25640 1 -- start start 2024-03-25T18:27:11.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.478+0000 7fe062e25640 10 monclient: init 2024-03-25T18:27:11.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.478+0000 7fe062e25640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:11.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.478+0000 7fe062e25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:11.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.478+0000 7fe062e25640 10 monclient(hunting): picked mon.noname-b con 0x7fe05c0a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:27:11.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.478+0000 7fe062e25640 10 monclient(hunting): picked mon.noname-c con 0x7fe05c13f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:27:11.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.478+0000 7fe062e25640 10 monclient(hunting): picked mon.noname-a con 0x7fe05c1430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:27:11.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.478+0000 7fe062e25640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05c142520 con 0x7fe05c1430c0 2024-03-25T18:27:11.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.478+0000 7fe062e25640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05c0b28a0 con 0x7fe05c13f710 2024-03-25T18:27:11.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.478+0000 7fe062e25640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05c0b2a30 con 0x7fe05c0a98f0 2024-03-25T18:27:11.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.478+0000 7fe062e25640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:11.481 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.478+0000 7fe062e25640 10 monclient(hunting): authenticate will time out at 7954.744809s 2024-03-25T18:27:11.490 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.488+0000 7fe060bc1640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7fe05c0a98f0 0x7fe05c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:51700/0 (socket says 172.21.15.122:51700) 2024-03-25T18:27:11.490 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.488+0000 7fe060bc1640 1 -- 172.21.15.122:0/1517062919 learned_addr learned my addr 172.21.15.122:0/1517062919 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:27:11.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.505+0000 7fe05b7fe640 1 -- 172.21.15.122:0/1517062919 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2627752886 0 0) 0x7fe05c142520 con 0x7fe05c1430c0 2024-03-25T18:27:11.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.505+0000 7fe05b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:11.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.505+0000 7fe05b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:11.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.505+0000 7fe05b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:11.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.505+0000 7fe05b7fe640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:27:11.507 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.505+0000 7fe05b7fe640 1 -- 172.21.15.122:0/1517062919 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe04c003520 con 0x7fe05c1430c0 2024-03-25T18:27:11.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.507+0000 7fe05b7fe640 1 -- 172.21.15.122:0/1517062919 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3208176150 0 0) 0x7fe05c0b2a30 con 0x7fe05c0a98f0 2024-03-25T18:27:11.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.507+0000 7fe05b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:11.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.507+0000 7fe05b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:11.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.507+0000 7fe05b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:11.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.507+0000 7fe05b7fe640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:27:11.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.507+0000 7fe05b7fe640 1 -- 172.21.15.122:0/1517062919 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe05c142520 con 0x7fe05c0a98f0 2024-03-25T18:27:11.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.508+0000 7fe05b7fe640 1 -- 172.21.15.122:0/1517062919 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715615658 0 0) 0x7fe05c0b28a0 con 0x7fe05c13f710 2024-03-25T18:27:11.509 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.508+0000 7fe05b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:11.510 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.508+0000 7fe05b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:11.510 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.508+0000 7fe05b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:11.510 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.508+0000 7fe05b7fe640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:27:11.510 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.508+0000 7fe05b7fe640 1 -- 172.21.15.122:0/1517062919 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe05c0b2a30 con 0x7fe05c13f710 2024-03-25T18:27:11.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.511+0000 7fe05b7fe640 1 -- 172.21.15.122:0/1517062919 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2284961986 0 0) 0x7fe04c003520 con 0x7fe05c1430c0 2024-03-25T18:27:11.513 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.512+0000 7fe05b7fe640 1 -- 172.21.15.122:0/1517062919 >> v1:172.21.15.122:6790/0 conn(0x7fe05c0a98f0 legacy=0x7fe05c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.512+0000 7fe05b7fe640 1 -- 172.21.15.122:0/1517062919 >> v1:172.21.15.204:6789/0 conn(0x7fe05c13f710 legacy=0x7fe05c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.512+0000 7fe05b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:11.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.512+0000 7fe05b7fe640 1 monclient: found mon.noname-a 2024-03-25T18:27:11.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.512+0000 7fe05b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:11.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.512+0000 7fe05b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:27:11.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.512+0000 7fe05b7fe640 1 -- 172.21.15.122:0/1517062919 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe05c0b44c0 con 0x7fe05c1430c0 2024-03-25T18:27:11.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.512+0000 7fe05b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:11.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.512+0000 7fe05b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:11.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.512+0000 7fe05b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:11.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.512+0000 7fe05b7fe640 1 -- 172.21.15.122:0/1517062919 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fe054001a10 con 0x7fe05c1430c0 2024-03-25T18:27:11.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.512+0000 7fe05b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:11.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.512+0000 7fe05b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:11.514 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.512+0000 7fe05b7fe640 10 monclient: dump: 2024-03-25T18:27:11.514 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:11.515 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:11.515 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:11.515 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:11.515 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:11.515 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:11.515 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:11.515 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:11.515 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:11.515 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:11.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.512+0000 7fe062e25640 5 monclient: authenticate success, global_id 4716 2024-03-25T18:27:11.515 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.512+0000 7fe062e25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:11.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.521+0000 7fe0613c2640 1 -- 172.21.15.122:0/1517062919 >> v1:172.21.15.122:6789/0 conn(0x7fe05c1430c0 legacy=0x7fe05c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T18:27:11.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.521+0000 7fe05b7fe640 1 -- 172.21.15.122:0/1517062919 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe054001b70 con 0x7fe05c1430c0 2024-03-25T18:27:11.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.521+0000 7fe0613c2640 1 -- 172.21.15.122:0/1517062919 >> v1:172.21.15.122:6789/0 conn(0x7fe05c1430c0 legacy=0x7fe05c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:27:11.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.521+0000 7fe05b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:11.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.521+0000 7fe0613c2640 1 --1- 172.21.15.122:0/1517062919 >> v1:172.21.15.122:6789/0 conn(0x7fe05c1430c0 0x7fe05c14dc30 :-1 s=OPENED pgs=81385 cs=1 l=1).handle_message read tag failed 2024-03-25T18:27:11.522 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.521+0000 7fe0613c2640 1 --1- 172.21.15.122:0/1517062919 >> v1:172.21.15.122:6789/0 conn(0x7fe05c1430c0 0x7fe05c14dc30 :-1 s=OPENED pgs=81385 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:27:11.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.521+0000 7fe062e25640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:11.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.521+0000 7fe062e25640 4 set_mon_vals no callback set 2024-03-25T18:27:11.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.521+0000 7fe05b7fe640 10 monclient: ms_handle_reset current mon v1:172.21.15.122:6789/0 2024-03-25T18:27:11.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.521+0000 7fe05b7fe640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:11.523 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.521+0000 7fe05b7fe640 1 -- 172.21.15.122:0/1517062919 >> v1:172.21.15.122:6789/0 conn(0x7fe05c1430c0 legacy=0x7fe05c14dc30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:27:11.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.521+0000 7fe062e25640 10 monclient: shutdown 2024-03-25T18:27:11.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.521+0000 7fe05b7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:11.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.521+0000 7fe05b7fe640 10 monclient(hunting): picked mon.a con 0x7fe04c003eb0 addr v1:172.21.15.122:6789/0 2024-03-25T18:27:11.525 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.521+0000 7fe05b7fe640 10 monclient(hunting): picked mon.b con 0x7fe05c13f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:27:11.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.522+0000 7fe05b7fe640 10 monclient(hunting): picked mon.c con 0x7fe05c0a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:27:11.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.522+0000 7fe05b7fe640 1 -- 172.21.15.122:0/1517062919 --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe04c003520 con 0x7fe04c003eb0 2024-03-25T18:27:11.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.522+0000 7fe05b7fe640 1 -- 172.21.15.122:0/1517062919 --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe05c0b28a0 con 0x7fe05c13f710 2024-03-25T18:27:11.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.522+0000 7fe05b7fe640 1 -- 172.21.15.122:0/1517062919 --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fe04c004540 con 0x7fe05c0a98f0 2024-03-25T18:27:11.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.522+0000 7fe05b7fe640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:11.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.522+0000 7fe05b7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.122:6789/0 2024-03-25T18:27:11.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.522+0000 7fe062e25640 20 monclient(hunting): shutdown discarding 1 pending message(s) 2024-03-25T18:27:11.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.522+0000 7fe062e25640 1 -- 172.21.15.122:0/1517062919 >> v1:172.21.15.122:6790/0 conn(0x7fe05c0a98f0 legacy=0x7fe04c00d9a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:27:11.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.522+0000 7fe062e25640 1 -- 172.21.15.122:0/1517062919 >> v1:172.21.15.204:6789/0 conn(0x7fe05c13f710 legacy=0x7fe04c004ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:27:11.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.522+0000 7fe062e25640 1 -- 172.21.15.122:0/1517062919 >> v1:172.21.15.122:6789/0 conn(0x7fe04c003eb0 legacy=0x7fe04c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:27:11.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.522+0000 7fe062e25640 1 -- 172.21.15.122:0/1517062919 shutdown_connections 2024-03-25T18:27:11.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.522+0000 7fe062e25640 1 -- 172.21.15.122:0/1517062919 >> 172.21.15.122:0/1517062919 conn(0x7fe05c0b7ea0 msgr2=0x7fe05c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:11.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.522+0000 7fe062e25640 1 -- 172.21.15.122:0/1517062919 shutdown_connections 2024-03-25T18:27:11.526 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.522+0000 7fe062e25640 1 -- 172.21.15.122:0/1517062919 wait complete. 2024-03-25T18:27:11.527 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.523+0000 7fe062e25640 10 monclient: build_initial_monmap 2024-03-25T18:27:11.527 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.523+0000 7fe062e25640 10 monclient: monmap: 2024-03-25T18:27:11.527 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:27:11.527 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:11.527 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:27:11.527 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:27:11.527 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:11.527 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:27:11.527 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:27:11.527 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:27:11.527 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:27:11.527 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:11.527 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.523+0000 7fe062e25640 1 librados: starting msgr at 2024-03-25T18:27:11.527 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.523+0000 7fe062e25640 1 librados: starting objecter 2024-03-25T18:27:11.527 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.523+0000 7fe062e25640 1 Processor -- start 2024-03-25T18:27:11.527 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.523+0000 7fe062e25640 1 -- start start 2024-03-25T18:27:11.528 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.523+0000 7fe062e25640 1 librados: setting wanted keys 2024-03-25T18:27:11.528 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.523+0000 7fe062e25640 1 librados: calling monclient init 2024-03-25T18:27:11.528 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.523+0000 7fe062e25640 10 monclient: init 2024-03-25T18:27:11.528 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.524+0000 7fe062e25640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:11.528 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.524+0000 7fe062e25640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:11.528 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.524+0000 7fe062e25640 10 monclient(hunting): picked mon.noname-c con 0x7fe04c003eb0 addr v1:172.21.15.122:6790/0 2024-03-25T18:27:11.528 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.524+0000 7fe062e25640 10 monclient(hunting): picked mon.noname-a con 0x7fe05c0a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:27:11.528 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.524+0000 7fe062e25640 10 monclient(hunting): picked mon.noname-b con 0x7fe05c13f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:27:11.528 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.524+0000 7fe062e25640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04c003520 con 0x7fe05c0a98f0 2024-03-25T18:27:11.528 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.524+0000 7fe062e25640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe05c0b28a0 con 0x7fe05c13f710 2024-03-25T18:27:11.528 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.524+0000 7fe062e25640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe04c004540 con 0x7fe04c003eb0 2024-03-25T18:27:11.528 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.524+0000 7fe062e25640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:11.528 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.524+0000 7fe062e25640 10 monclient(hunting): authenticate will time out at 7954.790391s 2024-03-25T18:27:11.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.537+0000 7fe05bfff640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe05c0a98f0 0x7fe05c138220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:37984/0 (socket says 172.21.15.122:37984) 2024-03-25T18:27:11.539 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.537+0000 7fe05bfff640 1 -- 172.21.15.122:0/2728457371 learned_addr learned my addr 172.21.15.122:0/2728457371 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:27:11.545 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.544+0000 7fe0597fa640 10 client.?.objecter ms_handle_connect 0x7fe05c0a98f0 2024-03-25T18:27:11.545 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.544+0000 7fe0597fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:11.548 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.547+0000 7fe0597fa640 10 client.?.objecter ms_handle_connect 0x7fe05c13f710 2024-03-25T18:27:11.548 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.547+0000 7fe0597fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:11.552 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.551+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959955576 0 0) 0x7fe04c003520 con 0x7fe05c0a98f0 2024-03-25T18:27:11.552 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.551+0000 7fe0597fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:11.552 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.551+0000 7fe0597fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:11.552 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.551+0000 7fe0597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:11.552 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.551+0000 7fe0597fa640 10 monclient(hunting): my global_id is 4719 2024-03-25T18:27:11.552 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.551+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe040004eb0 con 0x7fe05c0a98f0 2024-03-25T18:27:11.555 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.554+0000 7fe0597fa640 10 client.?.objecter ms_handle_connect 0x7fe04c003eb0 2024-03-25T18:27:11.555 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.554+0000 7fe0597fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:11.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.555+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3048973036 0 0) 0x7fe05c0b28a0 con 0x7fe05c13f710 2024-03-25T18:27:11.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.555+0000 7fe0597fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:11.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.555+0000 7fe0597fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:11.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.555+0000 7fe0597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:11.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.555+0000 7fe0597fa640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:27:11.556 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.555+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe04c003520 con 0x7fe05c13f710 2024-03-25T18:27:11.561 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.559+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607660179 0 0) 0x7fe04c004540 con 0x7fe04c003eb0 2024-03-25T18:27:11.561 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.559+0000 7fe0597fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:11.561 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.559+0000 7fe0597fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:11.561 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.559+0000 7fe0597fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:11.561 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.560+0000 7fe0597fa640 10 monclient(hunting): my global_id is 4673 2024-03-25T18:27:11.561 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.560+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe05c0b28a0 con 0x7fe04c003eb0 2024-03-25T18:27:11.564 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.562+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 476321493 0 0) 0x7fe04c003520 con 0x7fe05c13f710 2024-03-25T18:27:11.564 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.562+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe04c004540 con 0x7fe05c13f710 2024-03-25T18:27:11.564 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.562+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fe054001830 con 0x7fe05c13f710 2024-03-25T18:27:11.564 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.562+0000 7fe0597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:11.564 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.562+0000 7fe0597fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:11.564 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.562+0000 7fe0597fa640 10 monclient(hunting): dump: 2024-03-25T18:27:11.564 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:11.564 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:11.564 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:11.564 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:11.564 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:11.564 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:11.564 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:11.565 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:11.565 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:11.565 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:11.565 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.562+0000 7fe0597fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:11.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.566+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 730235340 0 0) 0x7fe040004eb0 con 0x7fe05c0a98f0 2024-03-25T18:27:11.568 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.566+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe04c003520 con 0x7fe05c0a98f0 2024-03-25T18:27:11.570 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3239036513 0 0) 0x7fe04c004540 con 0x7fe05c13f710 2024-03-25T18:27:11.570 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 >> v1:172.21.15.122:6790/0 conn(0x7fe04c003eb0 legacy=0x7fe05c1daf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.570 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 >> v1:172.21.15.122:6789/0 conn(0x7fe05c0a98f0 legacy=0x7fe05c138220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:11.570 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe0597fa640 10 monclient: _finish_hunting 0 2024-03-25T18:27:11.570 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe0597fa640 1 monclient: found mon.b 2024-03-25T18:27:11.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe0597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:11.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe0597fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:27:11.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04c0121b0 con 0x7fe05c13f710 2024-03-25T18:27:11.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe0597fa640 10 monclient: _finish_auth 0 2024-03-25T18:27:11.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe0597fa640 10 monclient: _check_auth_tickets 2024-03-25T18:27:11.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe0597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:11.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe062e25640 5 monclient: authenticate success, global_id 4705 2024-03-25T18:27:11.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe062e25640 10 monclient: _renew_subs 2024-03-25T18:27:11.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe062e25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:27:11.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe062e25640 1 -- 172.21.15.122:0/2728457371 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe05c0abac0 con 0x7fe05c13f710 2024-03-25T18:27:11.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe062e25640 10 client.4705.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:11.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe062e25640 10 monclient: _renew_subs 2024-03-25T18:27:11.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe062e25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:27:11.571 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe062e25640 1 -- 172.21.15.122:0/2728457371 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05c137760 con 0x7fe05c13f710 2024-03-25T18:27:11.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.568+0000 7fe062e25640 1 librados: init done 2024-03-25T18:27:11.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.570+0000 7fe062e25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:11.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.570+0000 7fe062e25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:11.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.570+0000 7fe062e25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:27:11.572 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.570+0000 7fe062e25640 1 -- 172.21.15.122:0/2728457371 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe05c0627b0 con 0x7fe05c13f710 2024-03-25T18:27:11.843 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.841+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0540019b0 con 0x7fe05c13f710 2024-03-25T18:27:11.843 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.841+0000 7fe0597fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:11.843 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.841+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fe054003dc0 con 0x7fe05c13f710 2024-03-25T18:27:11.843 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.841+0000 7fe0597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:11.843 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.841+0000 7fe0597fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:11.843 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.841+0000 7fe0597fa640 10 monclient: dump: 2024-03-25T18:27:11.843 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:11.843 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:11.844 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:11.844 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:11.844 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:11.844 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:11.844 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:11.844 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:11.844 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:11.844 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:11.844 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.841+0000 7fe05affd640 4 set_mon_vals no callback set 2024-03-25T18:27:11.845 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.842+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fe054003fa0 con 0x7fe05c13f710 2024-03-25T18:27:11.845 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.843+0000 7fe060bc1640 10 monclient: get_auth_request con 0x7fe04006e070 auth_method 0 2024-03-25T18:27:11.845 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.843+0000 7fe0597fa640 10 client.4705.objecter ms_handle_connect 0x7fe04006e070 2024-03-25T18:27:11.850 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.849+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1633957047 0 0) 0x7fe054054af0 con 0x7fe05c13f710 2024-03-25T18:27:11.850 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.849+0000 7fe0597fa640 10 client.4705.objecter ms_dispatch 0x7fe05c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:27:11.850 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.849+0000 7fe0597fa640 3 client.4705.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:27:11.850 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.849+0000 7fe0597fa640 3 client.4705.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:27:11.850 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.849+0000 7fe0597fa640 20 client.4705.objecter dump_active .. 0 homeless 2024-03-25T18:27:11.851 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.850+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe054050430 con 0x7fe05c13f710 2024-03-25T18:27:11.851 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.850+0000 7fe0597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:11.851 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:11.850+0000 7fe0597fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:12.112 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.110+0000 7fe062e25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:12.112 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.110+0000 7fe062e25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:12.112 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.110+0000 7fe062e25640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:27:12.112 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.110+0000 7fe062e25640 1 -- 172.21.15.122:0/2728457371 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe05c141a00 con 0x7fe05c13f710 2024-03-25T18:27:12.135 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.133+0000 7fe0597fa640 1 -- 172.21.15.122:0/2728457371 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1879996311) 0x7fe054052530 con 0x7fe05c13f710 2024-03-25T18:27:12.135 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.133+0000 7fe0597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:12.135 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.133+0000 7fe0597fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:12.135 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:27:12.135 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":11,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:26:32.649240+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:32.649240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:12.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.136+0000 7fe062e25640 10 librados: watch_flush enter 2024-03-25T18:27:12.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.136+0000 7fe062e25640 10 librados: watch_flush exit 2024-03-25T18:27:12.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.136+0000 7fe062e25640 20 client.4705.objecter shutdown clearing up homeless session... 2024-03-25T18:27:12.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.136+0000 7fe062e25640 10 client.4705.objecter successfully canceled tick 2024-03-25T18:27:12.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.136+0000 7fe062e25640 1 -- 172.21.15.122:0/2728457371 >> v1:172.21.15.204:6800/676228005 conn(0x7fe04006e070 legacy=0x7fe040070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:12.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.136+0000 7fe062e25640 10 monclient: shutdown 2024-03-25T18:27:12.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.136+0000 7fe062e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:12.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.136+0000 7fe062e25640 1 -- 172.21.15.122:0/2728457371 >> v1:172.21.15.204:6789/0 conn(0x7fe05c13f710 legacy=0x7fe05c1e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:12.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.136+0000 7fe062e25640 1 -- 172.21.15.122:0/2728457371 shutdown_connections 2024-03-25T18:27:12.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.136+0000 7fe062e25640 1 -- 172.21.15.122:0/2728457371 >> 172.21.15.122:0/2728457371 conn(0x7fe05c0b7ea0 msgr2=0x7fe05c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:12.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.136+0000 7fe062e25640 1 -- 172.21.15.122:0/2728457371 shutdown_connections 2024-03-25T18:27:12.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.136+0000 7fe062e25640 1 -- 172.21.15.122:0/2728457371 wait complete. 2024-03-25T18:27:12.139 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:12.136+0000 7fe062e25640 1 librados: shutdown 2024-03-25T18:27:12.149 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:17.863 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:11:11.605 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:17.869 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:11:11.611 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:17.908 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:11:11.650 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:17.920 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:11:11.666 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:17.929 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:11:11.674 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:17.947 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:11:11.689 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:17.953 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:11:11.695 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:17.956 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:11:11.701 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:17.987 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:11:11.729 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:18.957 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:19.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.072+0000 7fe61cdc9640 10 monclient: get_monmap_and_config 2024-03-25T18:27:19.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.072+0000 7fe61cdc9640 10 monclient: build_initial_monmap 2024-03-25T18:27:19.073 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.072+0000 7fe61cdc9640 10 monclient: monmap: 2024-03-25T18:27:19.073 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:27:19.073 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:19.073 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:27:19.073321+0000 2024-03-25T18:27:19.074 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:27:19.073321+0000 2024-03-25T18:27:19.074 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:19.074 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:19.074 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:27:19.074 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:27:19.074 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:27:19.074 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:19.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.073+0000 7fe61cdc9640 1 Processor -- start 2024-03-25T18:27:19.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.073+0000 7fe61cdc9640 1 -- start start 2024-03-25T18:27:19.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.073+0000 7fe61cdc9640 10 monclient: init 2024-03-25T18:27:19.079 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.073+0000 7fe61cdc9640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:19.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.073+0000 7fe61cdc9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:19.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.073+0000 7fe61cdc9640 10 monclient(hunting): picked mon.noname-b con 0x7fe6180b4d60 addr v1:172.21.15.122:6790/0 2024-03-25T18:27:19.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.073+0000 7fe61cdc9640 10 monclient(hunting): picked mon.noname-a con 0x7fe6180b1730 addr v1:172.21.15.122:6789/0 2024-03-25T18:27:19.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.073+0000 7fe61cdc9640 10 monclient(hunting): picked mon.noname-c con 0x7fe61814be90 addr v1:172.21.15.204:6789/0 2024-03-25T18:27:19.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.073+0000 7fe61cdc9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6180af940 con 0x7fe6180b1730 2024-03-25T18:27:19.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.073+0000 7fe61cdc9640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6180b0a70 con 0x7fe61814be90 2024-03-25T18:27:19.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.073+0000 7fe61cdc9640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6180b0d50 con 0x7fe6180b4d60 2024-03-25T18:27:19.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.073+0000 7fe61cdc9640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:19.080 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.073+0000 7fe61cdc9640 10 monclient(hunting): authenticate will time out at 7962.339941s 2024-03-25T18:27:19.088 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.087+0000 7fe615d9b640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7fe6180b1730 0x7fe6180b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:42680/0 (socket says 172.21.15.122:42680) 2024-03-25T18:27:19.088 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.087+0000 7fe615d9b640 1 -- 172.21.15.122:0/3744836225 learned_addr learned my addr 172.21.15.122:0/3744836225 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:27:19.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.098+0000 7fe61559a640 1 -- 172.21.15.122:0/3744836225 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2074075769 0 0) 0x7fe6180af940 con 0x7fe6180b1730 2024-03-25T18:27:19.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.098+0000 7fe61559a640 10 monclient(hunting): _negotiate 2024-03-25T18:27:19.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.098+0000 7fe61559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:19.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.098+0000 7fe61559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:19.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.098+0000 7fe61559a640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:27:19.100 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.098+0000 7fe61559a640 1 -- 172.21.15.122:0/3744836225 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe600003520 con 0x7fe6180b1730 2024-03-25T18:27:19.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61559a640 1 -- 172.21.15.122:0/3744836225 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3665854597 0 0) 0x7fe600003520 con 0x7fe6180b1730 2024-03-25T18:27:19.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61559a640 1 -- 172.21.15.122:0/3744836225 >> v1:172.21.15.122:6790/0 conn(0x7fe6180b4d60 legacy=0x7fe61813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:27:19.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61559a640 1 -- 172.21.15.122:0/3744836225 >> v1:172.21.15.204:6789/0 conn(0x7fe61814be90 legacy=0x7fe61814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:27:19.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61559a640 10 monclient: _finish_hunting 0 2024-03-25T18:27:19.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61559a640 1 monclient: found mon.noname-a 2024-03-25T18:27:19.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:19.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:27:19.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61559a640 1 -- 172.21.15.122:0/3744836225 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6180b1030 con 0x7fe6180b1730 2024-03-25T18:27:19.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61559a640 10 monclient: _finish_auth 0 2024-03-25T18:27:19.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61559a640 10 monclient: _check_auth_tickets 2024-03-25T18:27:19.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:19.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61559a640 1 -- 172.21.15.122:0/3744836225 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fe60c001690 con 0x7fe6180b1730 2024-03-25T18:27:19.117 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61cdc9640 5 monclient: authenticate success, global_id 4722 2024-03-25T18:27:19.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61cdc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:19.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:19.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:19.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61559a640 10 monclient: dump: 2024-03-25T18:27:19.118 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:19.118 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:19.118 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:19.118 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:19.118 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:19.118 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:19.118 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:19.118 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:19.118 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:19.118 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:19.118 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.115+0000 7fe61cdc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:19.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.127+0000 7fe61559a640 1 -- 172.21.15.122:0/3744836225 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe60c001860 con 0x7fe6180b1730 2024-03-25T18:27:19.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.127+0000 7fe61559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:19.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.127+0000 7fe61559a640 1 -- 172.21.15.122:0/3744836225 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fe60c001cb0 con 0x7fe6180b1730 2024-03-25T18:27:19.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.127+0000 7fe61cdc9640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:19.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.127+0000 7fe61cdc9640 4 set_mon_vals no callback set 2024-03-25T18:27:19.128 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.127+0000 7fe61559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:19.129 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.127+0000 7fe61559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:19.129 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.127+0000 7fe61559a640 10 monclient: dump: 2024-03-25T18:27:19.129 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:19.129 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:19.129 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:19.129 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:19.129 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:19.129 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:19.129 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:19.129 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:19.129 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:19.129 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:19.130 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.128+0000 7fe61cdc9640 10 monclient: shutdown 2024-03-25T18:27:19.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.128+0000 7fe61cdc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:19.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.128+0000 7fe61cdc9640 1 -- 172.21.15.122:0/3744836225 >> v1:172.21.15.122:6789/0 conn(0x7fe6180b1730 legacy=0x7fe6180b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:19.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.128+0000 7fe61cdc9640 1 -- 172.21.15.122:0/3744836225 shutdown_connections 2024-03-25T18:27:19.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.128+0000 7fe61cdc9640 1 -- 172.21.15.122:0/3744836225 >> 172.21.15.122:0/3744836225 conn(0x7fe6180617a0 msgr2=0x7fe618061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:19.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.128+0000 7fe61cdc9640 1 -- 172.21.15.122:0/3744836225 shutdown_connections 2024-03-25T18:27:19.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.128+0000 7fe61cdc9640 1 -- 172.21.15.122:0/3744836225 wait complete. 2024-03-25T18:27:19.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.129+0000 7fe61cdc9640 10 monclient: build_initial_monmap 2024-03-25T18:27:19.131 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.129+0000 7fe61cdc9640 10 monclient: monmap: 2024-03-25T18:27:19.131 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:27:19.131 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:19.131 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:27:19.131 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:27:19.131 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:19.131 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:27:19.132 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:27:19.132 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:27:19.132 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:27:19.132 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:19.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.129+0000 7fe61cdc9640 1 librados: starting msgr at 2024-03-25T18:27:19.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.129+0000 7fe61cdc9640 1 librados: starting objecter 2024-03-25T18:27:19.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.129+0000 7fe61cdc9640 1 Processor -- start 2024-03-25T18:27:19.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.129+0000 7fe61cdc9640 1 -- start start 2024-03-25T18:27:19.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.129+0000 7fe61cdc9640 1 librados: setting wanted keys 2024-03-25T18:27:19.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.129+0000 7fe61cdc9640 1 librados: calling monclient init 2024-03-25T18:27:19.132 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.129+0000 7fe61cdc9640 10 monclient: init 2024-03-25T18:27:19.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.130+0000 7fe61cdc9640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:19.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.130+0000 7fe61cdc9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:19.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.130+0000 7fe61cdc9640 10 monclient(hunting): picked mon.noname-c con 0x7fe6180b1730 addr v1:172.21.15.122:6790/0 2024-03-25T18:27:19.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.130+0000 7fe61cdc9640 10 monclient(hunting): picked mon.noname-b con 0x7fe6180b4d60 addr v1:172.21.15.204:6789/0 2024-03-25T18:27:19.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.130+0000 7fe61cdc9640 10 monclient(hunting): picked mon.noname-a con 0x7fe61814be90 addr v1:172.21.15.122:6789/0 2024-03-25T18:27:19.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.130+0000 7fe61cdc9640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe61813d0f0 con 0x7fe61814be90 2024-03-25T18:27:19.133 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.130+0000 7fe61cdc9640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe618140460 con 0x7fe6180b4d60 2024-03-25T18:27:19.134 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.130+0000 7fe61cdc9640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe618141b50 con 0x7fe6180b1730 2024-03-25T18:27:19.134 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.130+0000 7fe61cdc9640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:19.134 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.130+0000 7fe61cdc9640 10 monclient(hunting): authenticate will time out at 7962.396525s 2024-03-25T18:27:19.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.136+0000 7fe615d9b640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7fe6180b4d60 0x7fe618132d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:51818/0 (socket says 172.21.15.122:51818) 2024-03-25T18:27:19.138 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.136+0000 7fe615d9b640 1 -- 172.21.15.122:0/3472451511 learned_addr learned my addr 172.21.15.122:0/3472451511 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:27:19.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.141+0000 7fe606ffd640 10 client.?.objecter ms_handle_connect 0x7fe6180b4d60 2024-03-25T18:27:19.142 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.141+0000 7fe606ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:19.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.148+0000 7fe606ffd640 10 client.?.objecter ms_handle_connect 0x7fe61814be90 2024-03-25T18:27:19.149 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.148+0000 7fe606ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:19.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.150+0000 7fe606ffd640 10 client.?.objecter ms_handle_connect 0x7fe6180b1730 2024-03-25T18:27:19.151 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.150+0000 7fe606ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:19.158 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.156+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422869404 0 0) 0x7fe618140460 con 0x7fe6180b4d60 2024-03-25T18:27:19.158 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.156+0000 7fe606ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:19.158 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.156+0000 7fe606ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:19.158 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.156+0000 7fe606ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:19.158 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.156+0000 7fe606ffd640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:27:19.158 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.156+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5f8004eb0 con 0x7fe6180b4d60 2024-03-25T18:27:19.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.163+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2103433374 0 0) 0x7fe618141b50 con 0x7fe6180b1730 2024-03-25T18:27:19.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.163+0000 7fe606ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:19.164 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.163+0000 7fe606ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:19.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.163+0000 7fe606ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:19.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.163+0000 7fe606ffd640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:27:19.165 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.163+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe618140460 con 0x7fe6180b1730 2024-03-25T18:27:19.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.166+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 278681689 0 0) 0x7fe5f8004eb0 con 0x7fe6180b4d60 2024-03-25T18:27:19.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.166+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe618141b50 con 0x7fe6180b4d60 2024-03-25T18:27:19.167 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.166+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fe60c001b70 con 0x7fe6180b4d60 2024-03-25T18:27:19.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.166+0000 7fe606ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:19.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.166+0000 7fe606ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:19.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.166+0000 7fe606ffd640 10 monclient(hunting): dump: 2024-03-25T18:27:19.168 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:19.168 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:19.168 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:19.168 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:19.168 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:19.168 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:19.168 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:19.168 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:19.168 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:19.168 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:19.168 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.166+0000 7fe606ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:19.170 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.168+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 952411635 0 0) 0x7fe61813d0f0 con 0x7fe61814be90 2024-03-25T18:27:19.170 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.168+0000 7fe606ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:27:19.170 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.168+0000 7fe606ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:19.170 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.168+0000 7fe606ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:19.170 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.168+0000 7fe606ffd640 10 monclient(hunting): my global_id is 4725 2024-03-25T18:27:19.170 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.168+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5f8004eb0 con 0x7fe61814be90 2024-03-25T18:27:19.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4256420273 0 0) 0x7fe618141b50 con 0x7fe6180b4d60 2024-03-25T18:27:19.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 >> v1:172.21.15.122:6790/0 conn(0x7fe6180b1730 legacy=0x7fe618141440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:19.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 >> v1:172.21.15.122:6789/0 conn(0x7fe61814be90 legacy=0x7fe6181d7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:19.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe606ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:27:19.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe606ffd640 1 monclient: found mon.b 2024-03-25T18:27:19.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe606ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:19.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe606ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:27:19.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6181408c0 con 0x7fe6180b4d60 2024-03-25T18:27:19.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe606ffd640 10 monclient: _finish_auth 0 2024-03-25T18:27:19.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe606ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:27:19.179 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe606ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:19.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe61cdc9640 5 monclient: authenticate success, global_id 4708 2024-03-25T18:27:19.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe61cdc9640 10 monclient: _renew_subs 2024-03-25T18:27:19.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe61cdc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:27:19.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe61cdc9640 1 -- 172.21.15.122:0/3472451511 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe618143340 con 0x7fe6180b4d60 2024-03-25T18:27:19.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe61cdc9640 10 client.4708.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:19.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe61cdc9640 10 monclient: _renew_subs 2024-03-25T18:27:19.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe61cdc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:27:19.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe61cdc9640 1 -- 172.21.15.122:0/3472451511 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe618133610 con 0x7fe6180b4d60 2024-03-25T18:27:19.180 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.177+0000 7fe61cdc9640 1 librados: init done 2024-03-25T18:27:19.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.178+0000 7fe61cdc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:19.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.178+0000 7fe61cdc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:19.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.178+0000 7fe61cdc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:27:19.181 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.178+0000 7fe61cdc9640 1 -- 172.21.15.122:0/3472451511 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6180b22f0 con 0x7fe6180b4d60 2024-03-25T18:27:19.186 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.184+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe60c004bf0 con 0x7fe6180b4d60 2024-03-25T18:27:19.186 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.184+0000 7fe606ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:19.186 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.184+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7fe60c004d60 con 0x7fe6180b4d60 2024-03-25T18:27:19.186 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.184+0000 7fe606ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:19.186 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.184+0000 7fe606ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:19.186 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.184+0000 7fe606ffd640 10 monclient: dump: 2024-03-25T18:27:19.186 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:19.186 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:19.186 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:19.186 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:19.186 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:19.186 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:19.187 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:19.187 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:19.187 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:19.187 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:19.187 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.184+0000 7fe614d99640 4 set_mon_vals no callback set 2024-03-25T18:27:19.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.458+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7fe60c01ad30 con 0x7fe6180b4d60 2024-03-25T18:27:19.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.458+0000 7fe61659c640 10 monclient: get_auth_request con 0x7fe5f806ddb0 auth_method 0 2024-03-25T18:27:19.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.458+0000 7fe606ffd640 10 client.4708.objecter ms_handle_connect 0x7fe5f806ddb0 2024-03-25T18:27:19.462 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.460+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1633957047 0 0) 0x7fe60c054830 con 0x7fe6180b4d60 2024-03-25T18:27:19.462 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.460+0000 7fe606ffd640 10 client.4708.objecter ms_dispatch 0x7fe618000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:27:19.462 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.460+0000 7fe606ffd640 3 client.4708.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:27:19.462 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.460+0000 7fe606ffd640 3 client.4708.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:27:19.462 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.460+0000 7fe606ffd640 20 client.4708.objecter dump_active .. 0 homeless 2024-03-25T18:27:19.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.464+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe60c050340 con 0x7fe6180b4d60 2024-03-25T18:27:19.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.464+0000 7fe606ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:19.465 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.464+0000 7fe606ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:19.714 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.713+0000 7fe61cdc9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:19.715 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.713+0000 7fe61cdc9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:19.715 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.713+0000 7fe61cdc9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:27:19.715 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.713+0000 7fe61cdc9640 1 -- 172.21.15.122:0/3472451511 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe618140a50 con 0x7fe6180b4d60 2024-03-25T18:27:19.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.733+0000 7fe606ffd640 1 -- 172.21.15.122:0/3472451511 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1879996311) 0x7fe60c052270 con 0x7fe6180b4d60 2024-03-25T18:27:19.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.733+0000 7fe606ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:19.734 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.733+0000 7fe606ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:19.735 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:27:19.735 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":11,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:26:32.649240+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:32.649240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:19.737 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.735+0000 7fe61cdc9640 10 librados: watch_flush enter 2024-03-25T18:27:19.737 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.735+0000 7fe61cdc9640 10 librados: watch_flush exit 2024-03-25T18:27:19.737 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.735+0000 7fe61cdc9640 20 client.4708.objecter shutdown clearing up homeless session... 2024-03-25T18:27:19.737 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.735+0000 7fe61cdc9640 10 client.4708.objecter successfully canceled tick 2024-03-25T18:27:19.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.736+0000 7fe61cdc9640 1 -- 172.21.15.122:0/3472451511 >> v1:172.21.15.204:6800/676228005 conn(0x7fe5f806ddb0 legacy=0x7fe5f8070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:19.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.736+0000 7fe61cdc9640 10 monclient: shutdown 2024-03-25T18:27:19.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.736+0000 7fe61cdc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:19.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.736+0000 7fe61cdc9640 1 -- 172.21.15.122:0/3472451511 >> v1:172.21.15.204:6789/0 conn(0x7fe6180b4d60 legacy=0x7fe618132d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:19.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.736+0000 7fe61cdc9640 1 -- 172.21.15.122:0/3472451511 shutdown_connections 2024-03-25T18:27:19.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.736+0000 7fe61cdc9640 1 -- 172.21.15.122:0/3472451511 >> 172.21.15.122:0/3472451511 conn(0x7fe6180617a0 msgr2=0x7fe6180b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:19.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.736+0000 7fe61cdc9640 1 -- 172.21.15.122:0/3472451511 shutdown_connections 2024-03-25T18:27:19.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.736+0000 7fe61cdc9640 1 -- 172.21.15.122:0/3472451511 wait complete. 2024-03-25T18:27:19.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:19.736+0000 7fe61cdc9640 1 librados: shutdown 2024-03-25T18:27:19.749 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:26.556 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:26.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.669+0000 7f52331c6640 10 monclient: get_monmap_and_config 2024-03-25T18:27:26.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.669+0000 7f52331c6640 10 monclient: build_initial_monmap 2024-03-25T18:27:26.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.669+0000 7f52331c6640 10 monclient: monmap: 2024-03-25T18:27:26.671 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:27:26.671 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:26.671 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:27:26.670841+0000 2024-03-25T18:27:26.671 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:27:26.670841+0000 2024-03-25T18:27:26.671 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:26.671 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:26.671 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:27:26.672 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:27:26.672 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:27:26.672 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:26.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.670+0000 7f52331c6640 1 Processor -- start 2024-03-25T18:27:26.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.670+0000 7f52331c6640 1 -- start start 2024-03-25T18:27:26.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.670+0000 7f52331c6640 10 monclient: init 2024-03-25T18:27:26.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.671+0000 7f52331c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:26.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.671+0000 7f52331c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:26.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.671+0000 7f52331c6640 10 monclient(hunting): picked mon.noname-c con 0x7f522c0a98f0 addr v1:172.21.15.204:6789/0 2024-03-25T18:27:26.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.671+0000 7f52331c6640 10 monclient(hunting): picked mon.noname-a con 0x7f522c13f710 addr v1:172.21.15.122:6789/0 2024-03-25T18:27:26.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.671+0000 7f52331c6640 10 monclient(hunting): picked mon.noname-b con 0x7f522c1430c0 addr v1:172.21.15.122:6790/0 2024-03-25T18:27:26.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.671+0000 7f52331c6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522c142520 con 0x7f522c13f710 2024-03-25T18:27:26.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.671+0000 7f52331c6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522c0b28a0 con 0x7f522c0a98f0 2024-03-25T18:27:26.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.671+0000 7f52331c6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522c0b2a30 con 0x7f522c1430c0 2024-03-25T18:27:26.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.671+0000 7f52331c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:26.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.671+0000 7f52331c6640 10 monclient(hunting): authenticate will time out at 7969.937391s 2024-03-25T18:27:26.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.681+0000 7f5230f62640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f522c0a98f0 0x7f522c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:58660/0 (socket says 172.21.15.122:58660) 2024-03-25T18:27:26.682 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.681+0000 7f5230f62640 1 -- 172.21.15.122:0/939722518 learned_addr learned my addr 172.21.15.122:0/939722518 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:27:26.693 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.692+0000 7f522b7fe640 1 -- 172.21.15.122:0/939722518 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1121162205 0 0) 0x7f522c142520 con 0x7f522c13f710 2024-03-25T18:27:26.693 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.692+0000 7f522b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:26.693 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.692+0000 7f522b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:26.693 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.692+0000 7f522b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:26.694 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.692+0000 7f522b7fe640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:27:26.694 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.692+0000 7f522b7fe640 1 -- 172.21.15.122:0/939722518 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f521c003520 con 0x7f522c13f710 2024-03-25T18:27:26.699 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.698+0000 7f522b7fe640 1 -- 172.21.15.122:0/939722518 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 601776720 0 0) 0x7f522c0b2a30 con 0x7f522c1430c0 2024-03-25T18:27:26.700 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.698+0000 7f522b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:26.700 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.698+0000 7f522b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:26.700 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.698+0000 7f522b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:26.700 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.698+0000 7f522b7fe640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:27:26.700 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.698+0000 7f522b7fe640 1 -- 172.21.15.122:0/939722518 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f522c142520 con 0x7f522c1430c0 2024-03-25T18:27:26.709 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.707+0000 7f522b7fe640 1 -- 172.21.15.122:0/939722518 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172390100 0 0) 0x7f522c0b28a0 con 0x7f522c0a98f0 2024-03-25T18:27:26.709 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.707+0000 7f522b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:26.709 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.707+0000 7f522b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:26.709 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.707+0000 7f522b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:26.709 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.707+0000 7f522b7fe640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:27:26.709 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.707+0000 7f522b7fe640 1 -- 172.21.15.122:0/939722518 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f522c0b2a30 con 0x7f522c0a98f0 2024-03-25T18:27:26.717 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f522b7fe640 1 -- 172.21.15.122:0/939722518 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 972186492 0 0) 0x7f521c003520 con 0x7f522c13f710 2024-03-25T18:27:26.717 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f522b7fe640 1 -- 172.21.15.122:0/939722518 >> v1:172.21.15.122:6790/0 conn(0x7f522c1430c0 legacy=0x7f522c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.717 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f522b7fe640 1 -- 172.21.15.122:0/939722518 >> v1:172.21.15.204:6789/0 conn(0x7f522c0a98f0 legacy=0x7f522c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.717 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f522b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:26.717 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f522b7fe640 1 monclient: found mon.noname-a 2024-03-25T18:27:26.717 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f522b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:26.717 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f522b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:27:26.717 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f522b7fe640 1 -- 172.21.15.122:0/939722518 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c0b44c0 con 0x7f522c13f710 2024-03-25T18:27:26.717 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f522b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:26.717 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f522b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:26.717 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f522b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:26.717 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f522b7fe640 1 -- 172.21.15.122:0/939722518 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5218001690 con 0x7f522c13f710 2024-03-25T18:27:26.717 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f52331c6640 5 monclient: authenticate success, global_id 4728 2024-03-25T18:27:26.717 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f52331c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:26.718 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f522b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:26.718 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f522b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:26.718 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f522b7fe640 10 monclient: dump: 2024-03-25T18:27:26.718 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:26.718 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:26.718 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:26.718 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:26.718 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:26.718 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:26.718 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:26.718 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:26.718 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:26.718 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:26.718 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.715+0000 7f52331c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:26.722 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.721+0000 7f522b7fe640 1 -- 172.21.15.122:0/939722518 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52180017f0 con 0x7f522c13f710 2024-03-25T18:27:26.723 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.721+0000 7f522b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:26.723 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.721+0000 7f522b7fe640 1 -- 172.21.15.122:0/939722518 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5218001c00 con 0x7f522c13f710 2024-03-25T18:27:26.723 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.721+0000 7f52331c6640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:26.723 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.721+0000 7f52331c6640 4 set_mon_vals no callback set 2024-03-25T18:27:26.723 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.721+0000 7f522b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:26.723 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.721+0000 7f522b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:26.723 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.721+0000 7f522b7fe640 10 monclient: dump: 2024-03-25T18:27:26.723 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:26.723 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:26.723 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:26.723 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:26.723 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:26.723 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:26.723 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:26.723 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:26.723 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:26.723 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:26.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.722+0000 7f52331c6640 10 monclient: shutdown 2024-03-25T18:27:26.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.722+0000 7f52331c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:26.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.722+0000 7f52331c6640 1 -- 172.21.15.122:0/939722518 >> v1:172.21.15.122:6789/0 conn(0x7f522c13f710 legacy=0x7f522c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.722+0000 7f52331c6640 1 -- 172.21.15.122:0/939722518 shutdown_connections 2024-03-25T18:27:26.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.722+0000 7f52331c6640 1 -- 172.21.15.122:0/939722518 >> 172.21.15.122:0/939722518 conn(0x7f522c0b7ea0 msgr2=0x7f522c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:26.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.722+0000 7f52331c6640 1 -- 172.21.15.122:0/939722518 shutdown_connections 2024-03-25T18:27:26.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.722+0000 7f52331c6640 1 -- 172.21.15.122:0/939722518 wait complete. 2024-03-25T18:27:26.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.723+0000 7f52331c6640 10 monclient: build_initial_monmap 2024-03-25T18:27:26.725 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.723+0000 7f52331c6640 10 monclient: monmap: 2024-03-25T18:27:26.725 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:27:26.726 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:26.726 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:27:26.726 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:27:26.726 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:26.726 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:27:26.726 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:27:26.726 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:27:26.726 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:27:26.726 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:26.726 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.723+0000 7f52331c6640 1 librados: starting msgr at 2024-03-25T18:27:26.726 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.723+0000 7f52331c6640 1 librados: starting objecter 2024-03-25T18:27:26.726 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.723+0000 7f52331c6640 1 Processor -- start 2024-03-25T18:27:26.726 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.723+0000 7f52331c6640 1 -- start start 2024-03-25T18:27:26.726 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.723+0000 7f52331c6640 1 librados: setting wanted keys 2024-03-25T18:27:26.726 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.723+0000 7f52331c6640 1 librados: calling monclient init 2024-03-25T18:27:26.727 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.723+0000 7f52331c6640 10 monclient: init 2024-03-25T18:27:26.728 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.724+0000 7f52331c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:26.728 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.724+0000 7f52331c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:26.728 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.724+0000 7f52331c6640 10 monclient(hunting): picked mon.noname-c con 0x7f522c0a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:27:26.728 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.724+0000 7f52331c6640 10 monclient(hunting): picked mon.noname-b con 0x7f522c13f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:27:26.728 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.724+0000 7f52331c6640 10 monclient(hunting): picked mon.noname-a con 0x7f522c1430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:27:26.728 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.724+0000 7f52331c6640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522c0abd70 con 0x7f522c1430c0 2024-03-25T18:27:26.728 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.724+0000 7f52331c6640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522c1d7710 con 0x7f522c13f710 2024-03-25T18:27:26.728 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.724+0000 7f52331c6640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f522c1d8340 con 0x7f522c0a98f0 2024-03-25T18:27:26.728 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.724+0000 7f52331c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:26.728 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.724+0000 7f52331c6640 10 monclient(hunting): authenticate will time out at 7969.990616s 2024-03-25T18:27:26.735 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.734+0000 7f522bfff640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f522c13f710 0x7f522c1d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:58696/0 (socket says 172.21.15.122:58696) 2024-03-25T18:27:26.735 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.734+0000 7f522bfff640 1 -- 172.21.15.122:0/3694749806 learned_addr learned my addr 172.21.15.122:0/3694749806 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:27:26.735 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.734+0000 7f52297fa640 10 client.?.objecter ms_handle_connect 0x7f522c13f710 2024-03-25T18:27:26.735 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.734+0000 7f52297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:26.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.737+0000 7f52297fa640 10 client.?.objecter ms_handle_connect 0x7f522c0a98f0 2024-03-25T18:27:26.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.737+0000 7f52297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:26.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.737+0000 7f52297fa640 10 client.?.objecter ms_handle_connect 0x7f522c1430c0 2024-03-25T18:27:26.738 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.737+0000 7f52297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:26.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.749+0000 7f52297fa640 1 -- 172.21.15.122:0/3694749806 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2520321624 0 0) 0x7f522c0abd70 con 0x7f522c1430c0 2024-03-25T18:27:26.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.749+0000 7f52297fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:26.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.749+0000 7f52297fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:26.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.749+0000 7f52297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:26.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.749+0000 7f52297fa640 10 monclient(hunting): my global_id is 4731 2024-03-25T18:27:26.750 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.749+0000 7f52297fa640 1 -- 172.21.15.122:0/3694749806 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5210004eb0 con 0x7f522c1430c0 2024-03-25T18:27:26.822 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.821+0000 7f52297fa640 1 -- 172.21.15.122:0/3694749806 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2727700311 0 0) 0x7f5210004eb0 con 0x7f522c1430c0 2024-03-25T18:27:26.822 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.821+0000 7f52297fa640 1 -- 172.21.15.122:0/3694749806 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f522c0abd70 con 0x7f522c1430c0 2024-03-25T18:27:26.822 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.821+0000 7f52297fa640 1 -- 172.21.15.122:0/3694749806 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5224001b10 con 0x7f522c1430c0 2024-03-25T18:27:26.822 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.821+0000 7f52297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:26.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.821+0000 7f52297fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:26.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.821+0000 7f52297fa640 10 monclient(hunting): dump: 2024-03-25T18:27:26.823 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:26.823 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:26.823 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:26.823 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:26.823 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:26.823 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:26.823 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:26.823 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:26.823 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:26.823 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:26.823 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.821+0000 7f52297fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:26.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52297fa640 1 -- 172.21.15.122:0/3694749806 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 221758724 0 0) 0x7f522c0abd70 con 0x7f522c1430c0 2024-03-25T18:27:26.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52297fa640 1 -- 172.21.15.122:0/3694749806 >> v1:172.21.15.122:6790/0 conn(0x7f522c0a98f0 legacy=0x7f522c137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52297fa640 1 -- 172.21.15.122:0/3694749806 >> v1:172.21.15.204:6789/0 conn(0x7f522c13f710 legacy=0x7f522c1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:26.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52297fa640 10 monclient: _finish_hunting 0 2024-03-25T18:27:26.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52297fa640 1 monclient: found mon.a 2024-03-25T18:27:26.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:26.832 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52297fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:27:26.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52297fa640 1 -- 172.21.15.122:0/3694749806 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c1d84d0 con 0x7f522c1430c0 2024-03-25T18:27:26.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52297fa640 10 monclient: _finish_auth 0 2024-03-25T18:27:26.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52297fa640 10 monclient: _check_auth_tickets 2024-03-25T18:27:26.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:26.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52331c6640 5 monclient: authenticate success, global_id 4731 2024-03-25T18:27:26.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52331c6640 10 monclient: _renew_subs 2024-03-25T18:27:26.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52331c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:27:26.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52331c6640 1 -- 172.21.15.122:0/3694749806 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f522c1d41e0 con 0x7f522c1430c0 2024-03-25T18:27:26.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52331c6640 10 client.4731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:26.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52331c6640 10 monclient: _renew_subs 2024-03-25T18:27:26.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52331c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:27:26.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52331c6640 1 -- 172.21.15.122:0/3694749806 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f522c1d5150 con 0x7f522c1430c0 2024-03-25T18:27:26.833 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.830+0000 7f52331c6640 1 librados: init done 2024-03-25T18:27:26.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.832+0000 7f52331c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:26.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.832+0000 7f52331c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:26.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.832+0000 7f52331c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:27:26.834 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:26.832+0000 7f52331c6640 1 -- 172.21.15.122:0/3694749806 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f522c0627b0 con 0x7f522c1430c0 2024-03-25T18:27:27.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.095+0000 7f52297fa640 1 -- 172.21.15.122:0/3694749806 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5224002060 con 0x7f522c1430c0 2024-03-25T18:27:27.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.095+0000 7f52297fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:27.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.095+0000 7f52297fa640 1 -- 172.21.15.122:0/3694749806 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5224003b30 con 0x7f522c1430c0 2024-03-25T18:27:27.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.095+0000 7f52297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:27.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.095+0000 7f52297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:27.097 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.095+0000 7f52297fa640 10 monclient: dump: 2024-03-25T18:27:27.098 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:27.098 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:27.098 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:27.098 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:27.098 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:27.098 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:27.098 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:27.098 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:27.098 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:27.098 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:27.098 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.095+0000 7f522affd640 4 set_mon_vals no callback set 2024-03-25T18:27:27.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.096+0000 7f52297fa640 1 -- 172.21.15.122:0/3694749806 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f5224019b00 con 0x7f522c1430c0 2024-03-25T18:27:27.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.097+0000 7f5230f62640 10 monclient: get_auth_request con 0x7f521006dc50 auth_method 0 2024-03-25T18:27:27.099 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.097+0000 7f52297fa640 10 client.4731.objecter ms_handle_connect 0x7f521006dc50 2024-03-25T18:27:27.106 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.104+0000 7f52297fa640 1 -- 172.21.15.122:0/3694749806 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1633957047 0 0) 0x7f5224053880 con 0x7f522c1430c0 2024-03-25T18:27:27.106 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.104+0000 7f52297fa640 10 client.4731.objecter ms_dispatch 0x7f522c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:27:27.106 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.104+0000 7f52297fa640 3 client.4731.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:27:27.106 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.104+0000 7f52297fa640 3 client.4731.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:27:27.106 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.104+0000 7f52297fa640 20 client.4731.objecter dump_active .. 0 homeless 2024-03-25T18:27:27.106 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.104+0000 7f52297fa640 1 -- 172.21.15.122:0/3694749806 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f522404f1c0 con 0x7f522c1430c0 2024-03-25T18:27:27.106 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.104+0000 7f52297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:27.106 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.104+0000 7f52297fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:27.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.340+0000 7f52331c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:27.342 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.340+0000 7f52331c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:27.342 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.340+0000 7f52331c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:27:27.342 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.340+0000 7f52331c6640 1 -- 172.21.15.122:0/3694749806 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f522c1d7950 con 0x7f522c1430c0 2024-03-25T18:27:27.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.406+0000 7f52297fa640 1 -- 172.21.15.122:0/3694749806 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1879996311) 0x7f52240512c0 con 0x7f522c1430c0 2024-03-25T18:27:27.407 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.406+0000 7f52297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:27.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.406+0000 7f52297fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:27.408 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:27:27.408 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":11,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:26:32.649240+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:32.649240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:27.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.408+0000 7f52331c6640 10 librados: watch_flush enter 2024-03-25T18:27:27.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.408+0000 7f52331c6640 10 librados: watch_flush exit 2024-03-25T18:27:27.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.408+0000 7f52331c6640 20 client.4731.objecter shutdown clearing up homeless session... 2024-03-25T18:27:27.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.408+0000 7f52331c6640 10 client.4731.objecter successfully canceled tick 2024-03-25T18:27:27.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.408+0000 7f52331c6640 1 -- 172.21.15.122:0/3694749806 >> v1:172.21.15.204:6800/676228005 conn(0x7f521006dc50 legacy=0x7f5210070110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:27.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.408+0000 7f52331c6640 10 monclient: shutdown 2024-03-25T18:27:27.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.408+0000 7f52331c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:27.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.408+0000 7f52331c6640 1 -- 172.21.15.122:0/3694749806 >> v1:172.21.15.122:6789/0 conn(0x7f522c1430c0 legacy=0x7f522c1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:27.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.408+0000 7f52331c6640 1 -- 172.21.15.122:0/3694749806 shutdown_connections 2024-03-25T18:27:27.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.408+0000 7f52331c6640 1 -- 172.21.15.122:0/3694749806 >> 172.21.15.122:0/3694749806 conn(0x7f522c0b7ea0 msgr2=0x7f522c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:27.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.408+0000 7f52331c6640 1 -- 172.21.15.122:0/3694749806 shutdown_connections 2024-03-25T18:27:27.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.408+0000 7f52331c6640 1 -- 172.21.15.122:0/3694749806 wait complete. 2024-03-25T18:27:27.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:27.408+0000 7f52331c6640 1 librados: shutdown 2024-03-25T18:27:27.422 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:34.229 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:34.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.338+0000 7f5dcac1e640 10 monclient: get_monmap_and_config 2024-03-25T18:27:34.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.338+0000 7f5dcac1e640 10 monclient: build_initial_monmap 2024-03-25T18:27:34.340 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.338+0000 7f5dcac1e640 10 monclient: monmap: 2024-03-25T18:27:34.340 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:27:34.340 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:34.340 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:27:34.339848+0000 2024-03-25T18:27:34.340 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:27:34.339848+0000 2024-03-25T18:27:34.340 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:34.341 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:34.341 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:27:34.341 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:27:34.341 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:27:34.341 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:34.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.339+0000 7f5dcac1e640 1 Processor -- start 2024-03-25T18:27:34.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.340+0000 7f5dcac1e640 1 -- start start 2024-03-25T18:27:34.341 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.340+0000 7f5dcac1e640 10 monclient: init 2024-03-25T18:27:34.342 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.341+0000 7f5dcac1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:34.342 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.341+0000 7f5dcac1e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:27:34.342 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.341+0000 7f5dcac1e640 10 monclient(hunting): picked mon.noname-b con 0x7f5dc40a98f0 addr v1:172.21.15.122:6790/0 2024-03-25T18:27:34.343 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.341+0000 7f5dcac1e640 10 monclient(hunting): picked mon.noname-c con 0x7f5dc413f710 addr v1:172.21.15.204:6789/0 2024-03-25T18:27:34.343 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.341+0000 7f5dcac1e640 10 monclient(hunting): picked mon.noname-a con 0x7f5dc41430c0 addr v1:172.21.15.122:6789/0 2024-03-25T18:27:34.343 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.341+0000 7f5dcac1e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc4142520 con 0x7f5dc41430c0 2024-03-25T18:27:34.343 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.341+0000 7f5dcac1e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc40b28a0 con 0x7f5dc413f710 2024-03-25T18:27:34.343 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.341+0000 7f5dcac1e640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc40b2a30 con 0x7f5dc40a98f0 2024-03-25T18:27:34.343 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.341+0000 7f5dcac1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:34.343 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.341+0000 7f5dcac1e640 10 monclient(hunting): authenticate will time out at 7977.607371s 2024-03-25T18:27:34.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.354+0000 7f5dc91bb640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f5dc41430c0 0x7f5dc414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:38722/0 (socket says 172.21.15.122:38722) 2024-03-25T18:27:34.355 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.354+0000 7f5dc91bb640 1 -- 172.21.15.122:0/221229797 learned_addr learned my addr 172.21.15.122:0/221229797 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:27:34.378 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.377+0000 7f5dc37fe640 1 -- 172.21.15.122:0/221229797 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846859650 0 0) 0x7f5dc4142520 con 0x7f5dc41430c0 2024-03-25T18:27:34.379 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.377+0000 7f5dc37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:34.379 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.377+0000 7f5dc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:34.379 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.377+0000 7f5dc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:34.379 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.377+0000 7f5dc37fe640 10 monclient(hunting): my global_id is 4734 2024-03-25T18:27:34.379 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.377+0000 7f5dc37fe640 1 -- 172.21.15.122:0/221229797 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dac003520 con 0x7f5dc41430c0 2024-03-25T18:27:34.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.383+0000 7f5dc37fe640 1 -- 172.21.15.122:0/221229797 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 360283120 0 0) 0x7f5dc40b2a30 con 0x7f5dc40a98f0 2024-03-25T18:27:34.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.383+0000 7f5dc37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:34.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.383+0000 7f5dc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:34.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.383+0000 7f5dc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:34.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.383+0000 7f5dc37fe640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:27:34.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.383+0000 7f5dc37fe640 1 -- 172.21.15.122:0/221229797 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc4142520 con 0x7f5dc40a98f0 2024-03-25T18:27:34.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.384+0000 7f5dc37fe640 1 -- 172.21.15.122:0/221229797 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282236772 0 0) 0x7f5dc40b28a0 con 0x7f5dc413f710 2024-03-25T18:27:34.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.384+0000 7f5dc37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:34.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.384+0000 7f5dc37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:34.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.384+0000 7f5dc37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:34.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.384+0000 7f5dc37fe640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:27:34.386 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.384+0000 7f5dc37fe640 1 -- 172.21.15.122:0/221229797 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc40b2a30 con 0x7f5dc413f710 2024-03-25T18:27:34.393 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.391+0000 7f5dc37fe640 1 -- 172.21.15.122:0/221229797 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 843662194 0 0) 0x7f5dac003520 con 0x7f5dc41430c0 2024-03-25T18:27:34.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.391+0000 7f5dc37fe640 1 -- 172.21.15.122:0/221229797 >> v1:172.21.15.122:6790/0 conn(0x7f5dc40a98f0 legacy=0x7f5dc40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:34.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.392+0000 7f5dc37fe640 1 -- 172.21.15.122:0/221229797 >> v1:172.21.15.204:6789/0 conn(0x7f5dc413f710 legacy=0x7f5dc41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:34.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.392+0000 7f5dc37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:34.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.392+0000 7f5dc37fe640 1 monclient: found mon.noname-a 2024-03-25T18:27:34.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.392+0000 7f5dc37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:34.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.392+0000 7f5dc37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.122:6789/0 2024-03-25T18:27:34.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.392+0000 7f5dc37fe640 1 -- 172.21.15.122:0/221229797 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc40b44c0 con 0x7f5dc41430c0 2024-03-25T18:27:34.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.392+0000 7f5dc37fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:34.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.392+0000 7f5dc37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:34.394 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.392+0000 7f5dc37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:34.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.392+0000 7f5dc37fe640 1 -- 172.21.15.122:0/221229797 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5db4001690 con 0x7f5dc41430c0 2024-03-25T18:27:34.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.392+0000 7f5dcac1e640 5 monclient: authenticate success, global_id 4734 2024-03-25T18:27:34.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.392+0000 7f5dcac1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:34.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.392+0000 7f5dc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:34.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.392+0000 7f5dc37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:27:34.395 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.392+0000 7f5dc37fe640 10 monclient: dump: 2024-03-25T18:27:34.395 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:34.395 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:34.395 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:34.395 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:34.395 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:34.395 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:34.395 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:34.396 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:34.396 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:34.396 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:34.396 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.392+0000 7f5dcac1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:34.405 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.404+0000 7f5dc37fe640 1 -- 172.21.15.122:0/221229797 <== mon.0 v1:172.21.15.122:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5db40017f0 con 0x7f5dc41430c0 2024-03-25T18:27:34.405 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.404+0000 7f5dc37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:34.405 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.404+0000 7f5dc37fe640 1 -- 172.21.15.122:0/221229797 <== mon.0 v1:172.21.15.122:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5db4001c00 con 0x7f5dc41430c0 2024-03-25T18:27:34.405 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.404+0000 7f5dc37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:34.405 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.404+0000 7f5dc37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:34.405 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.404+0000 7f5dc37fe640 10 monclient: dump: 2024-03-25T18:27:34.405 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:34.406 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:34.406 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:34.406 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:34.406 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:34.406 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:34.406 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:34.406 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:34.406 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:34.406 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:34.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.404+0000 7f5dcac1e640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:34.406 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.404+0000 7f5dcac1e640 4 set_mon_vals no callback set 2024-03-25T18:27:34.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.404+0000 7f5dcac1e640 10 monclient: shutdown 2024-03-25T18:27:34.408 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.404+0000 7f5dcac1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:34.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.404+0000 7f5dcac1e640 1 -- 172.21.15.122:0/221229797 >> v1:172.21.15.122:6789/0 conn(0x7f5dc41430c0 legacy=0x7f5dc414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:34.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.404+0000 7f5dcac1e640 1 -- 172.21.15.122:0/221229797 shutdown_connections 2024-03-25T18:27:34.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.405+0000 7f5dcac1e640 1 -- 172.21.15.122:0/221229797 >> 172.21.15.122:0/221229797 conn(0x7f5dc40b7ea0 msgr2=0x7f5dc40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:34.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.405+0000 7f5dcac1e640 1 -- 172.21.15.122:0/221229797 shutdown_connections 2024-03-25T18:27:34.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.405+0000 7f5dcac1e640 1 -- 172.21.15.122:0/221229797 wait complete. 2024-03-25T18:27:34.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.405+0000 7f5dcac1e640 10 monclient: build_initial_monmap 2024-03-25T18:27:34.409 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.405+0000 7f5dcac1e640 10 monclient: monmap: 2024-03-25T18:27:34.409 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:27:34.409 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:34.409 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:27:34.409 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:27:34.409 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:34.409 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:27:34.409 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:27:34.410 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:27:34.410 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:27:34.410 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:34.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.406+0000 7f5dcac1e640 1 librados: starting msgr at 2024-03-25T18:27:34.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.406+0000 7f5dcac1e640 1 librados: starting objecter 2024-03-25T18:27:34.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.406+0000 7f5dcac1e640 1 Processor -- start 2024-03-25T18:27:34.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.406+0000 7f5dcac1e640 1 -- start start 2024-03-25T18:27:34.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.406+0000 7f5dcac1e640 1 librados: setting wanted keys 2024-03-25T18:27:34.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.406+0000 7f5dcac1e640 1 librados: calling monclient init 2024-03-25T18:27:34.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.406+0000 7f5dcac1e640 10 monclient: init 2024-03-25T18:27:34.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.406+0000 7f5dcac1e640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:34.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.407+0000 7f5dcac1e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:34.410 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.407+0000 7f5dcac1e640 10 monclient(hunting): picked mon.noname-a con 0x7f5dc40a98f0 addr v1:172.21.15.122:6789/0 2024-03-25T18:27:34.411 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.407+0000 7f5dcac1e640 10 monclient(hunting): picked mon.noname-c con 0x7f5dc413f710 addr v1:172.21.15.122:6790/0 2024-03-25T18:27:34.411 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.407+0000 7f5dcac1e640 10 monclient(hunting): picked mon.noname-b con 0x7f5dc41430c0 addr v1:172.21.15.204:6789/0 2024-03-25T18:27:34.411 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.407+0000 7f5dcac1e640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc414d520 con 0x7f5dc40a98f0 2024-03-25T18:27:34.411 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.407+0000 7f5dcac1e640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc41319b0 con 0x7f5dc41430c0 2024-03-25T18:27:34.411 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.407+0000 7f5dcac1e640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5dc406dea0 con 0x7f5dc413f710 2024-03-25T18:27:34.411 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.407+0000 7f5dcac1e640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:34.411 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.407+0000 7f5dcac1e640 10 monclient(hunting): authenticate will time out at 7977.673195s 2024-03-25T18:27:34.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.412+0000 7f5dc3fff640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f5dc413f710 0x7f5dc4130b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:34382/0 (socket says 172.21.15.122:34382) 2024-03-25T18:27:34.414 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.412+0000 7f5dc3fff640 1 -- 172.21.15.122:0/2761065853 learned_addr learned my addr 172.21.15.122:0/2761065853 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:27:34.417 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.416+0000 7f5dc17fa640 10 client.?.objecter ms_handle_connect 0x7f5dc413f710 2024-03-25T18:27:34.417 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.416+0000 7f5dc17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:34.431 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.430+0000 7f5dc17fa640 10 client.?.objecter ms_handle_connect 0x7f5dc41430c0 2024-03-25T18:27:34.432 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.430+0000 7f5dc17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:34.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.433+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260270497 0 0) 0x7f5dc406dea0 con 0x7f5dc413f710 2024-03-25T18:27:34.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.433+0000 7f5dc17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:34.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.433+0000 7f5dc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:34.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.433+0000 7f5dc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:34.434 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.433+0000 7f5dc17fa640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:27:34.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.433+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5db0004eb0 con 0x7f5dc413f710 2024-03-25T18:27:34.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.433+0000 7f5dc17fa640 10 client.?.objecter ms_handle_connect 0x7f5dc40a98f0 2024-03-25T18:27:34.435 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.433+0000 7f5dc17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:34.441 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.440+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048934387 0 0) 0x7f5dc414d520 con 0x7f5dc40a98f0 2024-03-25T18:27:34.441 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.440+0000 7f5dc17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:34.441 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.440+0000 7f5dc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:34.441 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.440+0000 7f5dc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:34.441 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.440+0000 7f5dc17fa640 10 monclient(hunting): my global_id is 4737 2024-03-25T18:27:34.441 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.440+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc406dea0 con 0x7f5dc40a98f0 2024-03-25T18:27:34.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.447+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2108417476 0 0) 0x7f5dc406dea0 con 0x7f5dc40a98f0 2024-03-25T18:27:34.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.447+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5dc414d520 con 0x7f5dc40a98f0 2024-03-25T18:27:34.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.447+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978661569 0 0) 0x7f5dc41319b0 con 0x7f5dc41430c0 2024-03-25T18:27:34.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.447+0000 7f5dc17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:27:34.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.447+0000 7f5dc17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:34.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.447+0000 7f5dc17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:34.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.447+0000 7f5dc17fa640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:27:34.449 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.447+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dc406dea0 con 0x7f5dc41430c0 2024-03-25T18:27:34.451 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.449+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1478774951 0 0) 0x7f5db0004eb0 con 0x7f5dc413f710 2024-03-25T18:27:34.451 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.449+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5dc41319b0 con 0x7f5dc413f710 2024-03-25T18:27:34.451 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.449+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5dbc001d90 con 0x7f5dc413f710 2024-03-25T18:27:34.451 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.449+0000 7f5dc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:34.451 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.449+0000 7f5dc17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:34.451 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.449+0000 7f5dc17fa640 10 monclient(hunting): dump: 2024-03-25T18:27:34.451 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:34.451 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:34.451 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:34.451 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:34.451 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:34.451 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:34.452 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:34.452 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:34.452 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:34.452 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:34.452 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.449+0000 7f5dc17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:34.452 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.451+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5db80019b0 con 0x7f5dc40a98f0 2024-03-25T18:27:34.453 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.451+0000 7f5dc17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:34.453 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.451+0000 7f5dc17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:34.453 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.451+0000 7f5dc17fa640 10 monclient(hunting): dump: 2024-03-25T18:27:34.453 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:34.453 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:34.453 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:34.453 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:34.453 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:34.453 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:34.453 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:34.453 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:34.453 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:34.453 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:34.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 <== mon.0 v1:172.21.15.122:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3507217051 0 0) 0x7f5dc414d520 con 0x7f5dc40a98f0 2024-03-25T18:27:34.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 >> v1:172.21.15.122:6790/0 conn(0x7f5dc413f710 legacy=0x7f5dc4130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:34.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 >> v1:172.21.15.204:6789/0 conn(0x7f5dc41430c0 legacy=0x7f5dc41312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:34.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dc17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:27:34.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dc17fa640 1 monclient: found mon.a 2024-03-25T18:27:34.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dc17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:34.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dc17fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:27:34.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 --> v1:172.21.15.122:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc414d790 con 0x7f5dc40a98f0 2024-03-25T18:27:34.458 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dc17fa640 10 monclient: _finish_auth 0 2024-03-25T18:27:34.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dc17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:27:34.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dc17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:34.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dcac1e640 5 monclient: authenticate success, global_id 4737 2024-03-25T18:27:34.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dcac1e640 10 monclient: _renew_subs 2024-03-25T18:27:34.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dcac1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:27:34.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dcac1e640 1 -- 172.21.15.122:0/2761065853 --> v1:172.21.15.122:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dc414da70 con 0x7f5dc40a98f0 2024-03-25T18:27:34.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dcac1e640 10 client.4737.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:34.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dcac1e640 10 monclient: _renew_subs 2024-03-25T18:27:34.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dcac1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:27:34.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dcac1e640 1 -- 172.21.15.122:0/2761065853 --> v1:172.21.15.122:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dc414bf00 con 0x7f5dc40a98f0 2024-03-25T18:27:34.459 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.456+0000 7f5dcac1e640 1 librados: init done 2024-03-25T18:27:34.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.457+0000 7f5dcac1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:34.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.457+0000 7f5dcac1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:34.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.457+0000 7f5dcac1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:27:34.460 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.457+0000 7f5dcac1e640 1 -- 172.21.15.122:0/2761065853 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dc40627b0 con 0x7f5dc40a98f0 2024-03-25T18:27:34.767 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.765+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 <== mon.0 v1:172.21.15.122:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5db8003610 con 0x7f5dc40a98f0 2024-03-25T18:27:34.767 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.765+0000 7f5dc17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:34.767 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.765+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 <== mon.0 v1:172.21.15.122:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f5db8003920 con 0x7f5dc40a98f0 2024-03-25T18:27:34.767 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.765+0000 7f5dc17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:34.767 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.765+0000 7f5dc17fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:34.768 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.765+0000 7f5dc17fa640 10 monclient: dump: 2024-03-25T18:27:34.768 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:34.768 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:34.768 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:34.768 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:34.768 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:34.768 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:34.768 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:34.768 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:34.768 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:34.768 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:34.768 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.765+0000 7f5dc2ffd640 4 set_mon_vals no callback set 2024-03-25T18:27:34.769 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.766+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 <== mon.0 v1:172.21.15.122:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f5db80198f0 con 0x7f5dc40a98f0 2024-03-25T18:27:34.769 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.767+0000 7f5dc3fff640 10 monclient: get_auth_request con 0x7f5db006df50 auth_method 0 2024-03-25T18:27:34.769 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.767+0000 7f5dc17fa640 10 client.4737.objecter ms_handle_connect 0x7f5db006df50 2024-03-25T18:27:34.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.781+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 <== mon.0 v1:172.21.15.122:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1633957047 0 0) 0x7f5db80536a0 con 0x7f5dc40a98f0 2024-03-25T18:27:34.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.781+0000 7f5dc17fa640 10 client.4737.objecter ms_dispatch 0x7f5dc4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:27:34.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.782+0000 7f5dc17fa640 3 client.4737.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:27:34.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.782+0000 7f5dc17fa640 3 client.4737.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:27:34.783 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.782+0000 7f5dc17fa640 20 client.4737.objecter dump_active .. 0 homeless 2024-03-25T18:27:34.784 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.782+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 <== mon.0 v1:172.21.15.122:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5db804efe0 con 0x7f5dc40a98f0 2024-03-25T18:27:34.784 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.782+0000 7f5dc17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:34.784 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:34.782+0000 7f5dc17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:35.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.005+0000 7f5dcac1e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:35.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.005+0000 7f5dcac1e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:35.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.005+0000 7f5dcac1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.122:6789/0 2024-03-25T18:27:35.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.005+0000 7f5dcac1e640 1 -- 172.21.15.122:0/2761065853 --> v1:172.21.15.122:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5dc413df70 con 0x7f5dc40a98f0 2024-03-25T18:27:35.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.028+0000 7f5dc17fa640 1 -- 172.21.15.122:0/2761065853 <== mon.0 v1:172.21.15.122:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1879996311) 0x7f5db80510e0 con 0x7f5dc40a98f0 2024-03-25T18:27:35.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.028+0000 7f5dc17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:35.029 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.028+0000 7f5dc17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:35.030 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:27:35.030 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":11,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:26:32.649240+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:32.649240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:35.031 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.030+0000 7f5dcac1e640 10 librados: watch_flush enter 2024-03-25T18:27:35.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.030+0000 7f5dcac1e640 10 librados: watch_flush exit 2024-03-25T18:27:35.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.030+0000 7f5dcac1e640 20 client.4737.objecter shutdown clearing up homeless session... 2024-03-25T18:27:35.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.030+0000 7f5dcac1e640 10 client.4737.objecter successfully canceled tick 2024-03-25T18:27:35.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.030+0000 7f5dcac1e640 1 -- 172.21.15.122:0/2761065853 >> v1:172.21.15.204:6800/676228005 conn(0x7f5db006df50 legacy=0x7f5db0070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:35.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.030+0000 7f5dcac1e640 10 monclient: shutdown 2024-03-25T18:27:35.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.030+0000 7f5dcac1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:35.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.030+0000 7f5dcac1e640 1 -- 172.21.15.122:0/2761065853 >> v1:172.21.15.122:6789/0 conn(0x7f5dc40a98f0 legacy=0x7f5dc413b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:35.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.030+0000 7f5dcac1e640 1 -- 172.21.15.122:0/2761065853 shutdown_connections 2024-03-25T18:27:35.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.030+0000 7f5dcac1e640 1 -- 172.21.15.122:0/2761065853 >> 172.21.15.122:0/2761065853 conn(0x7f5dc40b7ea0 msgr2=0x7f5dc413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:35.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.030+0000 7f5dcac1e640 1 -- 172.21.15.122:0/2761065853 shutdown_connections 2024-03-25T18:27:35.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.030+0000 7f5dcac1e640 1 -- 172.21.15.122:0/2761065853 wait complete. 2024-03-25T18:27:35.032 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:35.030+0000 7f5dcac1e640 1 librados: shutdown 2024-03-25T18:27:35.041 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:37.942 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:11:31.684 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:37.946 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:11:31.689 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:37.976 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:11:31.719 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:37.991 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:11:31.736 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.001 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:11:31.747 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.023 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:11:31.765 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.028 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:11:31.773 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.031 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:11:31.774 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:38.055 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:11:31.797 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:41.849 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:41.967 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.965+0000 7f849d71c640 10 monclient: get_monmap_and_config 2024-03-25T18:27:41.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.965+0000 7f849d71c640 10 monclient: build_initial_monmap 2024-03-25T18:27:41.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.965+0000 7f849d71c640 10 monclient: monmap: 2024-03-25T18:27:41.968 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:27:41.968 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:41.968 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:27:41.966848+0000 2024-03-25T18:27:41.968 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:27:41.966848+0000 2024-03-25T18:27:41.968 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:41.968 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:41.968 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:27:41.968 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:27:41.968 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:27:41.968 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:41.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.966+0000 7f849d71c640 1 Processor -- start 2024-03-25T18:27:41.968 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.966+0000 7f849d71c640 1 -- start start 2024-03-25T18:27:41.969 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.966+0000 7f849d71c640 10 monclient: init 2024-03-25T18:27:41.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.967+0000 7f849d71c640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:41.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.967+0000 7f849d71c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:27:41.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.967+0000 7f849d71c640 10 monclient(hunting): picked mon.noname-c con 0x7f84980705b0 addr v1:172.21.15.204:6789/0 2024-03-25T18:27:41.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.967+0000 7f849d71c640 10 monclient(hunting): picked mon.noname-b con 0x7f84981402d0 addr v1:172.21.15.122:6790/0 2024-03-25T18:27:41.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.967+0000 7f849d71c640 10 monclient(hunting): picked mon.noname-a con 0x7f8498149ae0 addr v1:172.21.15.122:6789/0 2024-03-25T18:27:41.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.967+0000 7f849d71c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84980718c0 con 0x7f8498149ae0 2024-03-25T18:27:41.970 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.967+0000 7f849d71c640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84980714c0 con 0x7f84980705b0 2024-03-25T18:27:41.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.967+0000 7f849d71c640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8498072b50 con 0x7f84981402d0 2024-03-25T18:27:41.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.967+0000 7f849d71c640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:41.971 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.967+0000 7f849d71c640 10 monclient(hunting): authenticate will time out at 7985.233224s 2024-03-25T18:27:41.978 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.977+0000 7f84977fe640 1 --1- >> v1:172.21.15.122:6789/0 conn(0x7f8498149ae0 0x7f849814bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6789/0 says I am v1:172.21.15.122:51740/0 (socket says 172.21.15.122:51740) 2024-03-25T18:27:41.978 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.977+0000 7f84977fe640 1 -- 172.21.15.122:0/523445578 learned_addr learned my addr 172.21.15.122:0/523445578 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:27:41.991 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.990+0000 7f8495ffb640 1 -- 172.21.15.122:0/523445578 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272120014 0 0) 0x7f84980714c0 con 0x7f84980705b0 2024-03-25T18:27:41.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.990+0000 7f8495ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:27:41.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.990+0000 7f8495ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:41.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.990+0000 7f8495ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:41.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.990+0000 7f8495ffb640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:27:41.992 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.990+0000 7f8495ffb640 1 -- 172.21.15.122:0/523445578 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8480003520 con 0x7f84980705b0 2024-03-25T18:27:41.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.994+0000 7f8495ffb640 1 -- 172.21.15.122:0/523445578 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3772750486 0 0) 0x7f8480003520 con 0x7f84980705b0 2024-03-25T18:27:41.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.994+0000 7f8495ffb640 1 -- 172.21.15.122:0/523445578 >> v1:172.21.15.122:6790/0 conn(0x7f84981402d0 legacy=0x7f8498140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:41.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.994+0000 7f8495ffb640 1 -- 172.21.15.122:0/523445578 >> v1:172.21.15.122:6789/0 conn(0x7f8498149ae0 legacy=0x7f849814bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:41.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.994+0000 7f8495ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:27:41.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.994+0000 7f8495ffb640 1 monclient: found mon.noname-c 2024-03-25T18:27:41.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.994+0000 7f8495ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:41.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.994+0000 7f8495ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:27:41.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.994+0000 7f8495ffb640 1 -- 172.21.15.122:0/523445578 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849813dd10 con 0x7f84980705b0 2024-03-25T18:27:41.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.994+0000 7f8495ffb640 10 monclient: _finish_auth 0 2024-03-25T18:27:41.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.994+0000 7f8495ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:27:41.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.994+0000 7f8495ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:41.996 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.994+0000 7f849d71c640 5 monclient: authenticate success, global_id 4723 2024-03-25T18:27:41.997 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.994+0000 7f849d71c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:41.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.997+0000 7f8495ffb640 1 -- 172.21.15.122:0/523445578 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f8488001690 con 0x7f84980705b0 2024-03-25T18:27:41.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.997+0000 7f8495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:41.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.997+0000 7f8495ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:41.998 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.997+0000 7f8495ffb640 10 monclient: dump: 2024-03-25T18:27:41.998 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:41.998 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:41.999 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:41.999 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:41.999 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:41.999 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:41.999 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:41.999 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:41.999 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:41.999 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:41.999 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:41.997+0000 7f849d71c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:42.001 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.000+0000 7f8495ffb640 1 -- 172.21.15.122:0/523445578 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8488001920 con 0x7f84980705b0 2024-03-25T18:27:42.001 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.000+0000 7f8495ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:42.001 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.000+0000 7f8495ffb640 1 -- 172.21.15.122:0/523445578 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f8488001c30 con 0x7f84980705b0 2024-03-25T18:27:42.001 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.000+0000 7f8495ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:42.001 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.000+0000 7f8495ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:42.001 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.000+0000 7f8495ffb640 10 monclient: dump: 2024-03-25T18:27:42.001 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:42.001 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:42.001 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:42.002 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:42.002 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:42.002 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:42.002 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:42.002 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:42.002 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:42.002 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:42.002 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.000+0000 7f849d71c640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:42.002 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.000+0000 7f849d71c640 4 set_mon_vals no callback set 2024-03-25T18:27:42.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.000+0000 7f849d71c640 10 monclient: shutdown 2024-03-25T18:27:42.003 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.000+0000 7f849d71c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:42.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.000+0000 7f849d71c640 1 -- 172.21.15.122:0/523445578 >> v1:172.21.15.204:6789/0 conn(0x7f84980705b0 legacy=0x7f849813d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:42.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.000+0000 7f849d71c640 1 -- 172.21.15.122:0/523445578 shutdown_connections 2024-03-25T18:27:42.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.000+0000 7f849d71c640 1 -- 172.21.15.122:0/523445578 >> 172.21.15.122:0/523445578 conn(0x7f84980b5d10 msgr2=0x7f84980b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:42.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.001+0000 7f849d71c640 1 -- 172.21.15.122:0/523445578 shutdown_connections 2024-03-25T18:27:42.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.001+0000 7f849d71c640 1 -- 172.21.15.122:0/523445578 wait complete. 2024-03-25T18:27:42.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.001+0000 7f849d71c640 10 monclient: build_initial_monmap 2024-03-25T18:27:42.004 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.001+0000 7f849d71c640 10 monclient: monmap: 2024-03-25T18:27:42.004 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:27:42.004 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:42.004 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:27:42.004 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:27:42.004 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:42.004 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:27:42.005 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:27:42.005 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:27:42.005 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:27:42.005 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:42.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.002+0000 7f849d71c640 1 librados: starting msgr at 2024-03-25T18:27:42.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.002+0000 7f849d71c640 1 librados: starting objecter 2024-03-25T18:27:42.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.002+0000 7f849d71c640 1 Processor -- start 2024-03-25T18:27:42.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.002+0000 7f849d71c640 1 -- start start 2024-03-25T18:27:42.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.002+0000 7f849d71c640 1 librados: setting wanted keys 2024-03-25T18:27:42.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.002+0000 7f849d71c640 1 librados: calling monclient init 2024-03-25T18:27:42.005 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.002+0000 7f849d71c640 10 monclient: init 2024-03-25T18:27:42.006 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.003+0000 7f849d71c640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:42.006 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.003+0000 7f849d71c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:27:42.006 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.003+0000 7f849d71c640 10 monclient(hunting): picked mon.noname-a con 0x7f84980705b0 addr v1:172.21.15.122:6789/0 2024-03-25T18:27:42.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.003+0000 7f849d71c640 10 monclient(hunting): picked mon.noname-b con 0x7f84981402d0 addr v1:172.21.15.204:6789/0 2024-03-25T18:27:42.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.003+0000 7f849d71c640 10 monclient(hunting): picked mon.noname-c con 0x7f8498149ae0 addr v1:172.21.15.122:6790/0 2024-03-25T18:27:42.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.003+0000 7f849d71c640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8498131b50 con 0x7f84980705b0 2024-03-25T18:27:42.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.003+0000 7f849d71c640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8498142140 con 0x7f84981402d0 2024-03-25T18:27:42.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.003+0000 7f849d71c640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8498142420 con 0x7f8498149ae0 2024-03-25T18:27:42.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.003+0000 7f849d71c640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:42.007 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.003+0000 7f849d71c640 10 monclient(hunting): authenticate will time out at 7985.269482s 2024-03-25T18:27:42.010 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.009+0000 7f84977fe640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f8498149ae0 0x7f84981e1190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:35016/0 (socket says 172.21.15.122:35016) 2024-03-25T18:27:42.011 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.009+0000 7f84977fe640 1 -- 172.21.15.122:0/3851267561 learned_addr learned my addr 172.21.15.122:0/3851267561 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:27:42.016 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.015+0000 7f847f7fe640 10 client.?.objecter ms_handle_connect 0x7f8498149ae0 2024-03-25T18:27:42.016 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.015+0000 7f847f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:42.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.023+0000 7f847f7fe640 10 client.?.objecter ms_handle_connect 0x7f84980705b0 2024-03-25T18:27:42.024 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.023+0000 7f847f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:42.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.026+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1077480861 0 0) 0x7f8498142420 con 0x7f8498149ae0 2024-03-25T18:27:42.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.026+0000 7f847f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:42.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.026+0000 7f847f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:42.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.026+0000 7f847f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:42.027 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.026+0000 7f847f7fe640 10 monclient(hunting): my global_id is 4694 2024-03-25T18:27:42.028 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.026+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8484004eb0 con 0x7f8498149ae0 2024-03-25T18:27:42.028 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.027+0000 7f847f7fe640 10 client.?.objecter ms_handle_connect 0x7f84981402d0 2024-03-25T18:27:42.028 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.027+0000 7f847f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:42.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.032+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1900440662 0 0) 0x7f8498131b50 con 0x7f84980705b0 2024-03-25T18:27:42.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.032+0000 7f847f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:42.033 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.032+0000 7f847f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:42.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.032+0000 7f847f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:42.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.032+0000 7f847f7fe640 10 monclient(hunting): my global_id is 4743 2024-03-25T18:27:42.034 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.032+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8498142420 con 0x7f84980705b0 2024-03-25T18:27:42.044 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.042+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2689706089 0 0) 0x7f8498142140 con 0x7f84981402d0 2024-03-25T18:27:42.044 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.042+0000 7f847f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:27:42.044 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.042+0000 7f847f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:42.044 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.042+0000 7f847f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:42.044 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.042+0000 7f847f7fe640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:27:42.044 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.042+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8498131b50 con 0x7f84981402d0 2024-03-25T18:27:42.045 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.044+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 209255028 0 0) 0x7f8484004eb0 con 0x7f8498149ae0 2024-03-25T18:27:42.045 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.044+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8498142140 con 0x7f8498149ae0 2024-03-25T18:27:42.046 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.044+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f8490001b50 con 0x7f8498149ae0 2024-03-25T18:27:42.046 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.044+0000 7f847f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:42.046 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.044+0000 7f847f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:42.046 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.044+0000 7f847f7fe640 10 monclient(hunting): dump: 2024-03-25T18:27:42.046 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:42.046 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:42.046 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:42.046 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:42.046 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:42.046 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:42.046 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:42.046 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:42.046 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:42.046 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:42.047 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.044+0000 7f847f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:42.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.054+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 <== mon.0 v1:172.21.15.122:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1783354182 0 0) 0x7f8498142420 con 0x7f84980705b0 2024-03-25T18:27:42.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.054+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 --> v1:172.21.15.122:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8484004eb0 con 0x7f84980705b0 2024-03-25T18:27:42.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.054+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 <== mon.0 v1:172.21.15.122:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f8488001650 con 0x7f84980705b0 2024-03-25T18:27:42.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.054+0000 7f847f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:42.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.054+0000 7f847f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:27:42.055 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.054+0000 7f847f7fe640 10 monclient(hunting): dump: 2024-03-25T18:27:42.055 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:42.055 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:42.055 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:42.056 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:42.056 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:42.056 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:42.056 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:42.056 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:42.056 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:42.056 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:42.068 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.066+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4241771862 0 0) 0x7f8498142140 con 0x7f8498149ae0 2024-03-25T18:27:42.068 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 >> v1:172.21.15.204:6789/0 conn(0x7f84981402d0 legacy=0x7f84981364b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:42.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 >> v1:172.21.15.122:6789/0 conn(0x7f84980705b0 legacy=0x7f8498143b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:42.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f847f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:27:42.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f847f7fe640 1 monclient: found mon.c 2024-03-25T18:27:42.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f847f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:42.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f847f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:27:42.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8498141290 con 0x7f8498149ae0 2024-03-25T18:27:42.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f847f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:27:42.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f847f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:27:42.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f847f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:42.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f849d71c640 5 monclient: authenticate success, global_id 4694 2024-03-25T18:27:42.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f849d71c640 10 monclient: _renew_subs 2024-03-25T18:27:42.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f849d71c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:27:42.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f849d71c640 1 -- 172.21.15.122:0/3851267561 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f849813fb50 con 0x7f8498149ae0 2024-03-25T18:27:42.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f849d71c640 10 client.4694.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:42.069 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f849d71c640 10 monclient: _renew_subs 2024-03-25T18:27:42.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f849d71c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:27:42.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f849d71c640 1 -- 172.21.15.122:0/3851267561 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f849813f810 con 0x7f8498149ae0 2024-03-25T18:27:42.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.067+0000 7f849d71c640 1 librados: init done 2024-03-25T18:27:42.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.068+0000 7f849d71c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:42.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.068+0000 7f849d71c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:42.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.068+0000 7f849d71c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:27:42.070 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.068+0000 7f849d71c640 1 -- 172.21.15.122:0/3851267561 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f849813fd90 con 0x7f8498149ae0 2024-03-25T18:27:42.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.375+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8490001fa0 con 0x7f8498149ae0 2024-03-25T18:27:42.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.375+0000 7f847f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:42.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.375+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f8490004d00 con 0x7f8498149ae0 2024-03-25T18:27:42.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.375+0000 7f847f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:42.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.375+0000 7f847f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:42.377 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.375+0000 7f847f7fe640 10 monclient: dump: 2024-03-25T18:27:42.377 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:42.378 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:42.378 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:42.378 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:42.378 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:42.378 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:42.378 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:42.378 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:42.378 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:42.378 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:42.378 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.375+0000 7f84957fa640 4 set_mon_vals no callback set 2024-03-25T18:27:42.379 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.376+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f8490004f20 con 0x7f8498149ae0 2024-03-25T18:27:42.379 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.377+0000 7f8496ffd640 10 monclient: get_auth_request con 0x7f848406df70 auth_method 0 2024-03-25T18:27:42.379 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.377+0000 7f847f7fe640 10 client.4694.objecter ms_handle_connect 0x7f848406df70 2024-03-25T18:27:42.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.382+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1633957047 0 0) 0x7f8490054a60 con 0x7f8498149ae0 2024-03-25T18:27:42.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.382+0000 7f847f7fe640 10 client.4694.objecter ms_dispatch 0x7f8498000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:27:42.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.382+0000 7f847f7fe640 3 client.4694.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:27:42.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.382+0000 7f847f7fe640 3 client.4694.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:27:42.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.383+0000 7f847f7fe640 20 client.4694.objecter dump_active .. 0 homeless 2024-03-25T18:27:42.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.383+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8490050570 con 0x7f8498149ae0 2024-03-25T18:27:42.384 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.383+0000 7f847f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:42.385 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.383+0000 7f847f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:42.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.643+0000 7f849d71c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:42.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.643+0000 7f849d71c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:42.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.643+0000 7f849d71c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:27:42.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.643+0000 7f849d71c640 1 -- 172.21.15.122:0/3851267561 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84980627b0 con 0x7f8498149ae0 2024-03-25T18:27:42.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.669+0000 7f847f7fe640 1 -- 172.21.15.122:0/3851267561 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1879996311) 0x7f84900524a0 con 0x7f8498149ae0 2024-03-25T18:27:42.670 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.669+0000 7f847f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:42.671 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.669+0000 7f847f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:42.671 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:27:42.671 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":11,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:26:32.649240+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:32.649240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:42.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.671+0000 7f849d71c640 10 librados: watch_flush enter 2024-03-25T18:27:42.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.671+0000 7f849d71c640 10 librados: watch_flush exit 2024-03-25T18:27:42.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.671+0000 7f849d71c640 20 client.4694.objecter shutdown clearing up homeless session... 2024-03-25T18:27:42.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.671+0000 7f849d71c640 10 client.4694.objecter successfully canceled tick 2024-03-25T18:27:42.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.672+0000 7f849d71c640 1 -- 172.21.15.122:0/3851267561 >> v1:172.21.15.204:6800/676228005 conn(0x7f848406df70 legacy=0x7f8484070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:42.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.672+0000 7f849d71c640 10 monclient: shutdown 2024-03-25T18:27:42.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.672+0000 7f849d71c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:42.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.672+0000 7f849d71c640 1 -- 172.21.15.122:0/3851267561 >> v1:172.21.15.122:6790/0 conn(0x7f8498149ae0 legacy=0x7f84981e1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:42.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.672+0000 7f849d71c640 1 -- 172.21.15.122:0/3851267561 shutdown_connections 2024-03-25T18:27:42.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.672+0000 7f849d71c640 1 -- 172.21.15.122:0/3851267561 >> 172.21.15.122:0/3851267561 conn(0x7f84980b5d10 msgr2=0x7f8498140730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:42.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.672+0000 7f849d71c640 1 -- 172.21.15.122:0/3851267561 shutdown_connections 2024-03-25T18:27:42.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.672+0000 7f849d71c640 1 -- 172.21.15.122:0/3851267561 wait complete. 2024-03-25T18:27:42.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:42.672+0000 7f849d71c640 1 librados: shutdown 2024-03-25T18:27:42.686 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:49.494 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:49.611 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.610+0000 7ffb6fb40640 10 monclient: get_monmap_and_config 2024-03-25T18:27:49.612 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.610+0000 7ffb6fb40640 10 monclient: build_initial_monmap 2024-03-25T18:27:49.612 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.610+0000 7ffb6fb40640 10 monclient: monmap: 2024-03-25T18:27:49.612 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:27:49.612 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:49.612 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:27:49.611624+0000 2024-03-25T18:27:49.612 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:27:49.611624+0000 2024-03-25T18:27:49.612 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:49.612 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:49.612 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:27:49.612 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:27:49.612 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:27:49.612 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:49.613 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.611+0000 7ffb6fb40640 1 Processor -- start 2024-03-25T18:27:49.613 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.611+0000 7ffb6fb40640 1 -- start start 2024-03-25T18:27:49.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.611+0000 7ffb6fb40640 10 monclient: init 2024-03-25T18:27:49.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.611+0000 7ffb6fb40640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:49.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.611+0000 7ffb6fb40640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:27:49.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.611+0000 7ffb6fb40640 10 monclient(hunting): picked mon.noname-c con 0x7ffb680b4d60 addr v1:172.21.15.204:6789/0 2024-03-25T18:27:49.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.611+0000 7ffb6fb40640 10 monclient(hunting): picked mon.noname-a con 0x7ffb680b1730 addr v1:172.21.15.122:6789/0 2024-03-25T18:27:49.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.611+0000 7ffb6fb40640 10 monclient(hunting): picked mon.noname-b con 0x7ffb6814be90 addr v1:172.21.15.122:6790/0 2024-03-25T18:27:49.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.612+0000 7ffb6fb40640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb680af940 con 0x7ffb680b1730 2024-03-25T18:27:49.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.612+0000 7ffb6fb40640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb680b0a70 con 0x7ffb680b4d60 2024-03-25T18:27:49.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.612+0000 7ffb6fb40640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb680b0d50 con 0x7ffb6814be90 2024-03-25T18:27:49.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.612+0000 7ffb6fb40640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:49.614 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.612+0000 7ffb6fb40640 10 monclient(hunting): authenticate will time out at 7992.878019s 2024-03-25T18:27:49.630 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.629+0000 7ffb6e0dd640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7ffb6814be90 0x7ffb6814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:39112/0 (socket says 172.21.15.122:39112) 2024-03-25T18:27:49.630 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.629+0000 7ffb6e0dd640 1 -- 172.21.15.122:0/1267107848 learned_addr learned my addr 172.21.15.122:0/1267107848 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:27:49.637 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.636+0000 7ffb6c8da640 1 -- 172.21.15.122:0/1267107848 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861057008 0 0) 0x7ffb680b0d50 con 0x7ffb6814be90 2024-03-25T18:27:49.637 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.636+0000 7ffb6c8da640 10 monclient(hunting): _negotiate 2024-03-25T18:27:49.637 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.636+0000 7ffb6c8da640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:49.637 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.636+0000 7ffb6c8da640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:49.637 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.636+0000 7ffb6c8da640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:27:49.637 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.636+0000 7ffb6c8da640 1 -- 172.21.15.122:0/1267107848 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb54003520 con 0x7ffb6814be90 2024-03-25T18:27:49.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.645+0000 7ffb6c8da640 1 -- 172.21.15.122:0/1267107848 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757622531 0 0) 0x7ffb680b0a70 con 0x7ffb680b4d60 2024-03-25T18:27:49.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.645+0000 7ffb6c8da640 10 monclient(hunting): _negotiate 2024-03-25T18:27:49.646 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.645+0000 7ffb6c8da640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:49.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.645+0000 7ffb6c8da640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:49.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.645+0000 7ffb6c8da640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:27:49.647 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.645+0000 7ffb6c8da640 1 -- 172.21.15.122:0/1267107848 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb680b0d50 con 0x7ffb680b4d60 2024-03-25T18:27:49.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6c8da640 1 -- 172.21.15.122:0/1267107848 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4160063545 0 0) 0x7ffb54003520 con 0x7ffb6814be90 2024-03-25T18:27:49.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6c8da640 1 -- 172.21.15.122:0/1267107848 >> v1:172.21.15.204:6789/0 conn(0x7ffb680b4d60 legacy=0x7ffb6813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:49.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6c8da640 1 -- 172.21.15.122:0/1267107848 >> v1:172.21.15.122:6789/0 conn(0x7ffb680b1730 legacy=0x7ffb680b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:49.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6c8da640 10 monclient: _finish_hunting 0 2024-03-25T18:27:49.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6c8da640 1 monclient: found mon.noname-b 2024-03-25T18:27:49.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6c8da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:49.650 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6c8da640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.122:6790/0 2024-03-25T18:27:49.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6c8da640 1 -- 172.21.15.122:0/1267107848 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb680b1030 con 0x7ffb6814be90 2024-03-25T18:27:49.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6c8da640 10 monclient: _finish_auth 0 2024-03-25T18:27:49.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6c8da640 10 monclient: _check_auth_tickets 2024-03-25T18:27:49.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6c8da640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:49.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6c8da640 1 -- 172.21.15.122:0/1267107848 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ffb60001a10 con 0x7ffb6814be90 2024-03-25T18:27:49.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6fb40640 5 monclient: authenticate success, global_id 4697 2024-03-25T18:27:49.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6fb40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:49.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6c8da640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:49.651 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6c8da640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:49.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6c8da640 10 monclient: dump: 2024-03-25T18:27:49.652 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:49.652 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:49.652 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:49.652 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:49.652 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:49.652 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:49.652 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:49.652 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:49.652 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:49.652 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:49.652 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.648+0000 7ffb6fb40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:49.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.656+0000 7ffb6c8da640 1 -- 172.21.15.122:0/1267107848 <== mon.2 v1:172.21.15.122:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb60001b70 con 0x7ffb6814be90 2024-03-25T18:27:49.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.656+0000 7ffb6c8da640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:49.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.656+0000 7ffb6c8da640 1 -- 172.21.15.122:0/1267107848 <== mon.2 v1:172.21.15.122:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ffb60001f80 con 0x7ffb6814be90 2024-03-25T18:27:49.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.656+0000 7ffb6c8da640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:49.657 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.656+0000 7ffb6c8da640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:49.658 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.656+0000 7ffb6c8da640 10 monclient: dump: 2024-03-25T18:27:49.658 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:49.658 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:49.658 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:49.658 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:49.658 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:49.658 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:49.658 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:49.658 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:49.658 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:49.658 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:49.659 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.656+0000 7ffb6fb40640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:49.659 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.656+0000 7ffb6fb40640 4 set_mon_vals no callback set 2024-03-25T18:27:49.659 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.657+0000 7ffb6fb40640 10 monclient: shutdown 2024-03-25T18:27:49.659 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.657+0000 7ffb6fb40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:49.659 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.657+0000 7ffb6fb40640 1 -- 172.21.15.122:0/1267107848 >> v1:172.21.15.122:6790/0 conn(0x7ffb6814be90 legacy=0x7ffb6814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:49.660 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.657+0000 7ffb6fb40640 1 -- 172.21.15.122:0/1267107848 shutdown_connections 2024-03-25T18:27:49.660 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.657+0000 7ffb6fb40640 1 -- 172.21.15.122:0/1267107848 >> 172.21.15.122:0/1267107848 conn(0x7ffb680617a0 msgr2=0x7ffb68061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:49.660 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.657+0000 7ffb6fb40640 1 -- 172.21.15.122:0/1267107848 shutdown_connections 2024-03-25T18:27:49.660 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.658+0000 7ffb6fb40640 1 -- 172.21.15.122:0/1267107848 wait complete. 2024-03-25T18:27:49.661 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.658+0000 7ffb6fb40640 10 monclient: build_initial_monmap 2024-03-25T18:27:49.661 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.658+0000 7ffb6fb40640 10 monclient: monmap: 2024-03-25T18:27:49.661 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:27:49.661 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:49.661 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:27:49.662 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:27:49.662 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:49.662 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:27:49.662 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:27:49.662 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:27:49.662 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:27:49.662 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:49.662 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.659+0000 7ffb6fb40640 1 librados: starting msgr at 2024-03-25T18:27:49.662 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.659+0000 7ffb6fb40640 1 librados: starting objecter 2024-03-25T18:27:49.662 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.659+0000 7ffb6fb40640 1 Processor -- start 2024-03-25T18:27:49.662 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.659+0000 7ffb6fb40640 1 -- start start 2024-03-25T18:27:49.662 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.659+0000 7ffb6fb40640 1 librados: setting wanted keys 2024-03-25T18:27:49.662 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.659+0000 7ffb6fb40640 1 librados: calling monclient init 2024-03-25T18:27:49.663 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.659+0000 7ffb6fb40640 10 monclient: init 2024-03-25T18:27:49.663 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.659+0000 7ffb6fb40640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:49.663 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.659+0000 7ffb6fb40640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:27:49.663 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.659+0000 7ffb6fb40640 10 monclient(hunting): picked mon.noname-c con 0x7ffb680b1730 addr v1:172.21.15.122:6790/0 2024-03-25T18:27:49.663 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.659+0000 7ffb6fb40640 10 monclient(hunting): picked mon.noname-a con 0x7ffb680b4d60 addr v1:172.21.15.122:6789/0 2024-03-25T18:27:49.663 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.659+0000 7ffb6fb40640 10 monclient(hunting): picked mon.noname-b con 0x7ffb6814be90 addr v1:172.21.15.204:6789/0 2024-03-25T18:27:49.663 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.659+0000 7ffb6fb40640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb68137b10 con 0x7ffb680b4d60 2024-03-25T18:27:49.663 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.659+0000 7ffb6fb40640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb681377e0 con 0x7ffb6814be90 2024-03-25T18:27:49.663 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.659+0000 7ffb6fb40640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb68137fa0 con 0x7ffb680b1730 2024-03-25T18:27:49.663 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.659+0000 7ffb6fb40640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:49.664 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.659+0000 7ffb6fb40640 10 monclient(hunting): authenticate will time out at 7992.925916s 2024-03-25T18:27:49.668 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.667+0000 7ffb6d8dc640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7ffb680b1730 0x7ffb68134eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:39144/0 (socket says 172.21.15.122:39144) 2024-03-25T18:27:49.668 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.667+0000 7ffb6d8dc640 1 -- 172.21.15.122:0/1324362533 learned_addr learned my addr 172.21.15.122:0/1324362533 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:27:49.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.672+0000 7ffb5e7fc640 10 client.?.objecter ms_handle_connect 0x7ffb6814be90 2024-03-25T18:27:49.673 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.672+0000 7ffb5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:49.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.673+0000 7ffb5e7fc640 10 client.?.objecter ms_handle_connect 0x7ffb680b1730 2024-03-25T18:27:49.674 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.673+0000 7ffb5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:49.677 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.676+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078333699 0 0) 0x7ffb681377e0 con 0x7ffb6814be90 2024-03-25T18:27:49.677 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.676+0000 7ffb5e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:49.677 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.676+0000 7ffb5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:49.678 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.676+0000 7ffb5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:49.678 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.676+0000 7ffb5e7fc640 10 monclient(hunting): my global_id is 4732 2024-03-25T18:27:49.678 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.676+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb4c004eb0 con 0x7ffb6814be90 2024-03-25T18:27:49.681 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.680+0000 7ffb5e7fc640 10 client.?.objecter ms_handle_connect 0x7ffb680b4d60 2024-03-25T18:27:49.681 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.680+0000 7ffb5e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:49.683 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.681+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933816276 0 0) 0x7ffb68137fa0 con 0x7ffb680b1730 2024-03-25T18:27:49.683 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.681+0000 7ffb5e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:49.683 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.681+0000 7ffb5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:49.683 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.681+0000 7ffb5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:49.683 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.681+0000 7ffb5e7fc640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:27:49.683 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.681+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb681377e0 con 0x7ffb680b1730 2024-03-25T18:27:49.686 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.684+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3704811957 0 0) 0x7ffb4c004eb0 con 0x7ffb6814be90 2024-03-25T18:27:49.686 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.684+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb68137fa0 con 0x7ffb6814be90 2024-03-25T18:27:49.686 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.684+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ffb60001e80 con 0x7ffb6814be90 2024-03-25T18:27:49.686 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.684+0000 7ffb5e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:49.686 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.684+0000 7ffb5e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:27:49.686 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.684+0000 7ffb5e7fc640 10 monclient(hunting): dump: 2024-03-25T18:27:49.686 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:49.686 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:49.687 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:49.687 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:49.687 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:49.687 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:49.687 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:49.687 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:49.687 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:49.687 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:49.687 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.684+0000 7ffb5e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:49.689 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.688+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 <== mon.0 v1:172.21.15.122:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326067669 0 0) 0x7ffb68137b10 con 0x7ffb680b4d60 2024-03-25T18:27:49.689 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.688+0000 7ffb5e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:49.690 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.688+0000 7ffb5e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:49.690 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.688+0000 7ffb5e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:49.690 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.688+0000 7ffb5e7fc640 10 monclient(hunting): my global_id is 4749 2024-03-25T18:27:49.690 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.688+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 --> v1:172.21.15.122:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb4c004eb0 con 0x7ffb680b4d60 2024-03-25T18:27:49.696 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 <== mon.1 v1:172.21.15.204:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4073348176 0 0) 0x7ffb68137fa0 con 0x7ffb6814be90 2024-03-25T18:27:49.696 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 >> v1:172.21.15.122:6790/0 conn(0x7ffb680b1730 legacy=0x7ffb68134eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:49.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 >> v1:172.21.15.122:6789/0 conn(0x7ffb680b4d60 legacy=0x7ffb681daed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:49.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb5e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:49.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb5e7fc640 1 monclient: found mon.b 2024-03-25T18:27:49.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb5e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:49.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb5e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:27:49.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb68142030 con 0x7ffb6814be90 2024-03-25T18:27:49.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb5e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:49.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb5e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:49.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb5e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:49.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb6fb40640 5 monclient: authenticate success, global_id 4732 2024-03-25T18:27:49.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb6fb40640 10 monclient: _renew_subs 2024-03-25T18:27:49.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb6fb40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:27:49.697 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb6fb40640 1 -- 172.21.15.122:0/1324362533 --> v1:172.21.15.204:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb68140840 con 0x7ffb6814be90 2024-03-25T18:27:49.698 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb6fb40640 10 client.4732.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:49.698 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb6fb40640 10 monclient: _renew_subs 2024-03-25T18:27:49.698 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb6fb40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:27:49.698 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb6fb40640 1 -- 172.21.15.122:0/1324362533 --> v1:172.21.15.204:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb681357a0 con 0x7ffb6814be90 2024-03-25T18:27:49.698 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.695+0000 7ffb6fb40640 1 librados: init done 2024-03-25T18:27:49.698 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.696+0000 7ffb6fb40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:49.699 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.696+0000 7ffb6fb40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:49.699 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.696+0000 7ffb6fb40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:27:49.699 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.696+0000 7ffb6fb40640 1 -- 172.21.15.122:0/1324362533 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb680627b0 con 0x7ffb6814be90 2024-03-25T18:27:49.919 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.917+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 <== mon.1 v1:172.21.15.204:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb60002040 con 0x7ffb6814be90 2024-03-25T18:27:49.919 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.917+0000 7ffb5e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:49.919 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.917+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 <== mon.1 v1:172.21.15.204:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7ffb60004fb0 con 0x7ffb6814be90 2024-03-25T18:27:49.919 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.917+0000 7ffb5e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:49.919 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.917+0000 7ffb5e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:49.919 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.917+0000 7ffb5e7fc640 10 monclient: dump: 2024-03-25T18:27:49.919 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:49.919 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:49.919 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:49.919 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:49.920 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:49.920 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:49.920 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:49.920 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:49.920 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:49.920 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:49.920 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.917+0000 7ffb5ffff640 4 set_mon_vals no callback set 2024-03-25T18:27:49.976 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.975+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 <== mon.1 v1:172.21.15.204:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7ffb6001af80 con 0x7ffb6814be90 2024-03-25T18:27:49.977 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.976+0000 7ffb6d8dc640 10 monclient: get_auth_request con 0x7ffb4c06de20 auth_method 0 2024-03-25T18:27:49.977 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.976+0000 7ffb5e7fc640 10 client.4732.objecter ms_handle_connect 0x7ffb4c06de20 2024-03-25T18:27:49.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.978+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 <== mon.1 v1:172.21.15.204:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1633957047 0 0) 0x7ffb600549c0 con 0x7ffb6814be90 2024-03-25T18:27:49.979 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.978+0000 7ffb5e7fc640 10 client.4732.objecter ms_dispatch 0x7ffb68000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:27:49.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.978+0000 7ffb5e7fc640 3 client.4732.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:27:49.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.978+0000 7ffb5e7fc640 3 client.4732.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:27:49.980 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.978+0000 7ffb5e7fc640 20 client.4732.objecter dump_active .. 0 homeless 2024-03-25T18:27:49.982 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.981+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 <== mon.1 v1:172.21.15.204:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffb600504d0 con 0x7ffb6814be90 2024-03-25T18:27:49.982 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.981+0000 7ffb5e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:49.982 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:49.981+0000 7ffb5e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:50.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.233+0000 7ffb6fb40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:50.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.233+0000 7ffb6fb40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:50.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.233+0000 7ffb6fb40640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.204:6789/0 2024-03-25T18:27:50.235 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.233+0000 7ffb6fb40640 1 -- 172.21.15.122:0/1324362533 --> v1:172.21.15.204:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffb68137cc0 con 0x7ffb6814be90 2024-03-25T18:27:50.264 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.262+0000 7ffb5e7fc640 1 -- 172.21.15.122:0/1324362533 <== mon.1 v1:172.21.15.204:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1879996311) 0x7ffb68137cc0 con 0x7ffb6814be90 2024-03-25T18:27:50.264 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.262+0000 7ffb5e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:50.264 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.262+0000 7ffb5e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:50.264 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:27:50.265 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":11,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:26:32.649240+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:32.649240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:50.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.264+0000 7ffb6fb40640 10 librados: watch_flush enter 2024-03-25T18:27:50.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.264+0000 7ffb6fb40640 10 librados: watch_flush exit 2024-03-25T18:27:50.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.264+0000 7ffb6fb40640 20 client.4732.objecter shutdown clearing up homeless session... 2024-03-25T18:27:50.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.264+0000 7ffb6fb40640 10 client.4732.objecter successfully canceled tick 2024-03-25T18:27:50.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.264+0000 7ffb6fb40640 1 -- 172.21.15.122:0/1324362533 >> v1:172.21.15.204:6800/676228005 conn(0x7ffb4c06de20 legacy=0x7ffb4c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:50.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.264+0000 7ffb6fb40640 10 monclient: shutdown 2024-03-25T18:27:50.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.264+0000 7ffb6fb40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:50.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.264+0000 7ffb6fb40640 1 -- 172.21.15.122:0/1324362533 >> v1:172.21.15.204:6789/0 conn(0x7ffb6814be90 legacy=0x7ffb681e3270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:50.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.264+0000 7ffb6fb40640 1 -- 172.21.15.122:0/1324362533 shutdown_connections 2024-03-25T18:27:50.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.264+0000 7ffb6fb40640 1 -- 172.21.15.122:0/1324362533 >> 172.21.15.122:0/1324362533 conn(0x7ffb680617a0 msgr2=0x7ffb6813c680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:50.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.265+0000 7ffb6fb40640 1 -- 172.21.15.122:0/1324362533 shutdown_connections 2024-03-25T18:27:50.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.265+0000 7ffb6fb40640 1 -- 172.21.15.122:0/1324362533 wait complete. 2024-03-25T18:27:50.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:50.265+0000 7ffb6fb40640 1 librados: shutdown 2024-03-25T18:27:50.277 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:57.085 DEBUG:teuthology.orchestra.run.smithi122:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:27:57.201 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.200+0000 7f729c093640 10 monclient: get_monmap_and_config 2024-03-25T18:27:57.202 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.200+0000 7f729c093640 10 monclient: build_initial_monmap 2024-03-25T18:27:57.202 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.200+0000 7f729c093640 10 monclient: monmap: 2024-03-25T18:27:57.202 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:27:57.202 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:57.202 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:27:57.201612+0000 2024-03-25T18:27:57.202 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:27:57.201612+0000 2024-03-25T18:27:57.202 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:57.202 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:57.202 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:27:57.202 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-c 2024-03-25T18:27:57.202 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-b 2024-03-25T18:27:57.202 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:57.204 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.201+0000 7f729c093640 1 Processor -- start 2024-03-25T18:27:57.204 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.201+0000 7f729c093640 1 -- start start 2024-03-25T18:27:57.204 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.201+0000 7f729c093640 10 monclient: init 2024-03-25T18:27:57.204 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.202+0000 7f729c093640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:57.204 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.202+0000 7f729c093640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:57.204 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.202+0000 7f729c093640 10 monclient(hunting): picked mon.noname-a con 0x7f72940b4d60 addr v1:172.21.15.122:6789/0 2024-03-25T18:27:57.204 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.202+0000 7f729c093640 10 monclient(hunting): picked mon.noname-b con 0x7f72940b1730 addr v1:172.21.15.122:6790/0 2024-03-25T18:27:57.204 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.202+0000 7f729c093640 10 monclient(hunting): picked mon.noname-c con 0x7f729414be90 addr v1:172.21.15.204:6789/0 2024-03-25T18:27:57.204 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.202+0000 7f729c093640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72940af940 con 0x7f72940b4d60 2024-03-25T18:27:57.204 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.202+0000 7f729c093640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72940b0a70 con 0x7f729414be90 2024-03-25T18:27:57.204 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.202+0000 7f729c093640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72940b0d50 con 0x7f72940b1730 2024-03-25T18:27:57.204 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.202+0000 7f729c093640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:57.205 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.202+0000 7f729c093640 10 monclient(hunting): authenticate will time out at 8000.468472s 2024-03-25T18:27:57.210 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.206+0000 7f729a630640 1 --1- >> v1:172.21.15.204:6789/0 conn(0x7f729414be90 0x7f729414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.204:6789/0 says I am v1:172.21.15.122:48894/0 (socket says 172.21.15.122:48894) 2024-03-25T18:27:57.210 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.207+0000 7f729a630640 1 -- 172.21.15.122:0/4018149489 learned_addr learned my addr 172.21.15.122:0/4018149489 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:27:57.218 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.217+0000 7f7298e2d640 1 -- 172.21.15.122:0/4018149489 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520694861 0 0) 0x7f72940b0a70 con 0x7f729414be90 2024-03-25T18:27:57.218 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.217+0000 7f7298e2d640 10 monclient(hunting): _negotiate 2024-03-25T18:27:57.218 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.217+0000 7f7298e2d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:57.218 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.217+0000 7f7298e2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:57.218 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.217+0000 7f7298e2d640 10 monclient(hunting): my global_id is 4735 2024-03-25T18:27:57.219 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.217+0000 7f7298e2d640 1 -- 172.21.15.122:0/4018149489 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7280003520 con 0x7f729414be90 2024-03-25T18:27:57.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.223+0000 7f7298e2d640 1 -- 172.21.15.122:0/4018149489 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4003825131 0 0) 0x7f7280003520 con 0x7f729414be90 2024-03-25T18:27:57.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.223+0000 7f7298e2d640 1 -- 172.21.15.122:0/4018149489 >> v1:172.21.15.122:6790/0 conn(0x7f72940b1730 legacy=0x7f72940b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:57.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.223+0000 7f7298e2d640 1 -- 172.21.15.122:0/4018149489 >> v1:172.21.15.122:6789/0 conn(0x7f72940b4d60 legacy=0x7f729413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:57.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.223+0000 7f7298e2d640 10 monclient: _finish_hunting 0 2024-03-25T18:27:57.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.223+0000 7f7298e2d640 1 monclient: found mon.noname-c 2024-03-25T18:27:57.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.223+0000 7f7298e2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:57.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.223+0000 7f7298e2d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.204:6789/0 2024-03-25T18:27:57.225 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.223+0000 7f7298e2d640 1 -- 172.21.15.122:0/4018149489 --> v1:172.21.15.204:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72940b1030 con 0x7f729414be90 2024-03-25T18:27:57.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.223+0000 7f7298e2d640 10 monclient: _finish_auth 0 2024-03-25T18:27:57.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.223+0000 7f7298e2d640 10 monclient: _check_auth_tickets 2024-03-25T18:27:57.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.223+0000 7f7298e2d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:57.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.223+0000 7f7298e2d640 1 -- 172.21.15.122:0/4018149489 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f728c001a10 con 0x7f729414be90 2024-03-25T18:27:57.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.223+0000 7f7298e2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:57.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.223+0000 7f7298e2d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:57.226 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.223+0000 7f7298e2d640 10 monclient: dump: 2024-03-25T18:27:57.226 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:57.226 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:57.226 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:57.226 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:57.226 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:57.226 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:57.226 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:57.227 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:57.227 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:57.227 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:57.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.223+0000 7f729c093640 5 monclient: authenticate success, global_id 4735 2024-03-25T18:27:57.227 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.223+0000 7f729c093640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:27:57.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.233+0000 7f7298e2d640 1 -- 172.21.15.122:0/4018149489 <== mon.1 v1:172.21.15.204:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f728c001b70 con 0x7f729414be90 2024-03-25T18:27:57.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.233+0000 7f7298e2d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:57.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.233+0000 7f7298e2d640 1 -- 172.21.15.122:0/4018149489 <== mon.1 v1:172.21.15.204:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f728c001f80 con 0x7f729414be90 2024-03-25T18:27:57.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.233+0000 7f729c093640 10 monclient: get_monmap_and_config success 2024-03-25T18:27:57.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.233+0000 7f729c093640 4 set_mon_vals no callback set 2024-03-25T18:27:57.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.233+0000 7f7298e2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:57.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.233+0000 7f7298e2d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:57.234 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.233+0000 7f7298e2d640 10 monclient: dump: 2024-03-25T18:27:57.235 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:57.235 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:57.235 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:57.235 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:57.235 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:57.235 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:57.235 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:57.235 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:57.235 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:57.235 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:57.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.233+0000 7f729c093640 10 monclient: shutdown 2024-03-25T18:27:57.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.234+0000 7f729c093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:57.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.234+0000 7f729c093640 1 -- 172.21.15.122:0/4018149489 >> v1:172.21.15.204:6789/0 conn(0x7f729414be90 legacy=0x7f729414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:57.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.234+0000 7f729c093640 1 -- 172.21.15.122:0/4018149489 shutdown_connections 2024-03-25T18:27:57.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.234+0000 7f729c093640 1 -- 172.21.15.122:0/4018149489 >> 172.21.15.122:0/4018149489 conn(0x7f72940617a0 msgr2=0x7f7294061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:57.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.234+0000 7f729c093640 1 -- 172.21.15.122:0/4018149489 shutdown_connections 2024-03-25T18:27:57.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.234+0000 7f729c093640 1 -- 172.21.15.122:0/4018149489 wait complete. 2024-03-25T18:27:57.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.235+0000 7f729c093640 10 monclient: build_initial_monmap 2024-03-25T18:27:57.237 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.235+0000 7f729c093640 10 monclient: monmap: 2024-03-25T18:27:57.237 INFO:teuthology.misc.health.smithi122.stderr:epoch 0 2024-03-25T18:27:57.237 INFO:teuthology.misc.health.smithi122.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:27:57.237 INFO:teuthology.misc.health.smithi122.stderr:last_changed 0.000000 2024-03-25T18:27:57.238 INFO:teuthology.misc.health.smithi122.stderr:created 0.000000 2024-03-25T18:27:57.238 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 0 (unknown) 2024-03-25T18:27:57.238 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 1 2024-03-25T18:27:57.238 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.noname-a 2024-03-25T18:27:57.238 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.noname-b 2024-03-25T18:27:57.238 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.noname-c 2024-03-25T18:27:57.238 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:57.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.235+0000 7f729c093640 1 librados: starting msgr at 2024-03-25T18:27:57.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.235+0000 7f729c093640 1 librados: starting objecter 2024-03-25T18:27:57.238 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.235+0000 7f729c093640 1 Processor -- start 2024-03-25T18:27:57.239 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.235+0000 7f729c093640 1 -- start start 2024-03-25T18:27:57.239 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.235+0000 7f729c093640 1 librados: setting wanted keys 2024-03-25T18:27:57.239 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.235+0000 7f729c093640 1 librados: calling monclient init 2024-03-25T18:27:57.239 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.235+0000 7f729c093640 10 monclient: init 2024-03-25T18:27:57.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.236+0000 7f729c093640 10 monclient: _reopen_session rank -1 2024-03-25T18:27:57.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.236+0000 7f729c093640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:27:57.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.236+0000 7f729c093640 10 monclient(hunting): picked mon.noname-a con 0x7f72940b1730 addr v1:172.21.15.122:6789/0 2024-03-25T18:27:57.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.236+0000 7f729c093640 10 monclient(hunting): picked mon.noname-c con 0x7f72940b4d60 addr v1:172.21.15.122:6790/0 2024-03-25T18:27:57.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.236+0000 7f729c093640 10 monclient(hunting): picked mon.noname-b con 0x7f729414be90 addr v1:172.21.15.204:6789/0 2024-03-25T18:27:57.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.236+0000 7f729c093640 1 -- --> v1:172.21.15.122:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7294140e70 con 0x7f72940b1730 2024-03-25T18:27:57.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.236+0000 7f729c093640 1 -- --> v1:172.21.15.204:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7294141fc0 con 0x7f729414be90 2024-03-25T18:27:57.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.236+0000 7f729c093640 1 -- --> v1:172.21.15.122:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7294143200 con 0x7f72940b4d60 2024-03-25T18:27:57.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.236+0000 7f729c093640 10 monclient(hunting): _renew_subs 2024-03-25T18:27:57.240 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.236+0000 7f729c093640 10 monclient(hunting): authenticate will time out at 8000.502562s 2024-03-25T18:27:57.256 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.255+0000 7f729962e640 1 --1- >> v1:172.21.15.122:6790/0 conn(0x7f72940b4d60 0x7f72941423e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.122:6790/0 says I am v1:172.21.15.122:44046/0 (socket says 172.21.15.122:44046) 2024-03-25T18:27:57.256 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.255+0000 7f729962e640 1 -- 172.21.15.122:0/1076987834 learned_addr learned my addr 172.21.15.122:0/1076987834 (peer_addr_for_me v1:172.21.15.122:0/0) 2024-03-25T18:27:57.257 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.256+0000 7f728a7fc640 10 client.?.objecter ms_handle_connect 0x7f72940b4d60 2024-03-25T18:27:57.257 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.256+0000 7f728a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:57.262 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.261+0000 7f728a7fc640 10 client.?.objecter ms_handle_connect 0x7f729414be90 2024-03-25T18:27:57.262 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.261+0000 7f728a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:57.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.265+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 <== mon.1 v1:172.21.15.204:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4150522989 0 0) 0x7f7294141fc0 con 0x7f729414be90 2024-03-25T18:27:57.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.265+0000 7f728a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:57.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.265+0000 7f728a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:57.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.265+0000 7f728a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:57.266 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.265+0000 7f728a7fc640 10 monclient(hunting): my global_id is 4738 2024-03-25T18:27:57.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.265+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 --> v1:172.21.15.204:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7278004eb0 con 0x7f729414be90 2024-03-25T18:27:57.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.266+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 <== mon.2 v1:172.21.15.122:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332156600 0 0) 0x7f7294143200 con 0x7f72940b4d60 2024-03-25T18:27:57.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.266+0000 7f728a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:27:57.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.266+0000 7f728a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:27:57.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.266+0000 7f728a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:27:57.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.266+0000 7f728a7fc640 10 monclient(hunting): my global_id is 4706 2024-03-25T18:27:57.267 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.266+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 --> v1:172.21.15.122:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7294141fc0 con 0x7f72940b4d60 2024-03-25T18:27:57.270 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.269+0000 7f728a7fc640 10 client.?.objecter ms_handle_connect 0x7f72940b1730 2024-03-25T18:27:57.270 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.269+0000 7f728a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:27:57.278 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.277+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 <== mon.2 v1:172.21.15.122:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3655989903 0 0) 0x7f7294141fc0 con 0x7f72940b4d60 2024-03-25T18:27:57.278 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.277+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 --> v1:172.21.15.122:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7294143200 con 0x7f72940b4d60 2024-03-25T18:27:57.278 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.277+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 <== mon.2 v1:172.21.15.122:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f7284001990 con 0x7f72940b4d60 2024-03-25T18:27:57.278 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.277+0000 7f728a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:57.278 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.277+0000 7f728a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:27:57.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.277+0000 7f728a7fc640 10 monclient(hunting): dump: 2024-03-25T18:27:57.279 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:57.279 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:57.279 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:57.279 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:57.279 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:57.279 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:57.279 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:57.279 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:57.279 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:57.279 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:57.279 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.277+0000 7f728a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:27:57.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.278+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 <== mon.1 v1:172.21.15.204:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 622968306 0 0) 0x7f7278004eb0 con 0x7f729414be90 2024-03-25T18:27:57.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.278+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 --> v1:172.21.15.204:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7294141fc0 con 0x7f729414be90 2024-03-25T18:27:57.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.278+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 <== mon.1 v1:172.21.15.204:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f728c001e80 con 0x7f729414be90 2024-03-25T18:27:57.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.278+0000 7f728a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:57.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.278+0000 7f728a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:27:57.280 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.278+0000 7f728a7fc640 10 monclient(hunting): dump: 2024-03-25T18:27:57.280 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:57.280 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:57.280 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:57.280 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:57.280 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:57.281 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:57.281 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:57.281 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:57.281 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:57.281 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:57.288 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 <== mon.2 v1:172.21.15.122:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3830907495 0 0) 0x7f7294143200 con 0x7f72940b4d60 2024-03-25T18:27:57.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 >> v1:172.21.15.204:6789/0 conn(0x7f729414be90 legacy=0x7f7294142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:57.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 >> v1:172.21.15.122:6789/0 conn(0x7f72940b1730 legacy=0x7f7294140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:57.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f728a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:27:57.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f728a7fc640 1 monclient: found mon.c 2024-03-25T18:27:57.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f728a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:27:57.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f728a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:27:57.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 --> v1:172.21.15.122:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7294145770 con 0x7f72940b4d60 2024-03-25T18:27:57.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f728a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:27:57.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f728a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:27:57.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f728a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:27:57.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f729c093640 5 monclient: authenticate success, global_id 4706 2024-03-25T18:27:57.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f729c093640 10 monclient: _renew_subs 2024-03-25T18:27:57.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f729c093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:27:57.289 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f729c093640 1 -- 172.21.15.122:0/1076987834 --> v1:172.21.15.122:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f729413e250 con 0x7f72940b4d60 2024-03-25T18:27:57.290 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f729c093640 10 client.4706.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:27:57.290 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f729c093640 10 monclient: _renew_subs 2024-03-25T18:27:57.290 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f729c093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:27:57.290 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f729c093640 1 -- 172.21.15.122:0/1076987834 --> v1:172.21.15.122:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7294144f90 con 0x7f72940b4d60 2024-03-25T18:27:57.290 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.287+0000 7f729c093640 1 librados: init done 2024-03-25T18:27:57.290 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.288+0000 7f729c093640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:57.291 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.288+0000 7f729c093640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:57.291 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.288+0000 7f729c093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:27:57.291 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.288+0000 7f729c093640 1 -- 172.21.15.122:0/1076987834 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7294141de0 con 0x7f72940b4d60 2024-03-25T18:27:57.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.585+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 <== mon.2 v1:172.21.15.122:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7284003610 con 0x7f72940b4d60 2024-03-25T18:27:57.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.586+0000 7f728a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:27:57.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.586+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 <== mon.2 v1:172.21.15.122:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164814613 0 0) 0x7f72840039e0 con 0x7f72940b4d60 2024-03-25T18:27:57.587 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.586+0000 7f728a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:27:57.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.586+0000 7f728a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:27:57.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.586+0000 7f728a7fc640 10 monclient: dump: 2024-03-25T18:27:57.588 INFO:teuthology.misc.health.smithi122.stderr:epoch 1 2024-03-25T18:27:57.588 INFO:teuthology.misc.health.smithi122.stderr:fsid b4b19611-696c-4b9f-a4ec-3ee2f6191d98 2024-03-25T18:27:57.588 INFO:teuthology.misc.health.smithi122.stderr:last_changed 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:57.588 INFO:teuthology.misc.health.smithi122.stderr:created 2024-03-25T18:15:33.375703+0000 2024-03-25T18:27:57.588 INFO:teuthology.misc.health.smithi122.stderr:min_mon_release 19 (squid) 2024-03-25T18:27:57.588 INFO:teuthology.misc.health.smithi122.stderr:election_strategy: 3 2024-03-25T18:27:57.588 INFO:teuthology.misc.health.smithi122.stderr:0: v1:172.21.15.122:6789/0 mon.a 2024-03-25T18:27:57.588 INFO:teuthology.misc.health.smithi122.stderr:1: v1:172.21.15.204:6789/0 mon.b 2024-03-25T18:27:57.588 INFO:teuthology.misc.health.smithi122.stderr:2: v1:172.21.15.122:6790/0 mon.c 2024-03-25T18:27:57.588 INFO:teuthology.misc.health.smithi122.stderr: 2024-03-25T18:27:57.588 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.586+0000 7f728bfff640 4 set_mon_vals no callback set 2024-03-25T18:27:57.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.586+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 <== mon.2 v1:172.21.15.122:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 996029198 0 0) 0x7f7284003c00 con 0x7f72940b4d60 2024-03-25T18:27:57.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.587+0000 7f7299e2f640 10 monclient: get_auth_request con 0x7f727806dd90 auth_method 0 2024-03-25T18:27:57.589 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.587+0000 7f728a7fc640 10 client.4706.objecter ms_handle_connect 0x7f727806dd90 2024-03-25T18:27:57.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.592+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 <== mon.2 v1:172.21.15.122:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 1633957047 0 0) 0x7f7284054760 con 0x7f72940b4d60 2024-03-25T18:27:57.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.592+0000 7f728a7fc640 10 client.4706.objecter ms_dispatch 0x7f7294000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:27:57.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.592+0000 7f728a7fc640 3 client.4706.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:27:57.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.592+0000 7f728a7fc640 3 client.4706.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:27:57.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.592+0000 7f728a7fc640 20 client.4706.objecter dump_active .. 0 homeless 2024-03-25T18:27:57.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.592+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 <== mon.2 v1:172.21.15.122:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7284050270 con 0x7f72940b4d60 2024-03-25T18:27:57.594 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.593+0000 7f728a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:27:57.595 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.593+0000 7f728a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:27:57.856 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.854+0000 7f729c093640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:57.856 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.854+0000 7f729c093640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:57.856 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.854+0000 7f729c093640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.122:6790/0 2024-03-25T18:27:57.856 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.854+0000 7f729c093640 1 -- 172.21.15.122:0/1076987834 --> v1:172.21.15.122:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7294145900 con 0x7f72940b4d60 2024-03-25T18:27:57.899 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.897+0000 7f728a7fc640 1 -- 172.21.15.122:0/1076987834 <== mon.2 v1:172.21.15.122:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1879996311) 0x7f72840521a0 con 0x7f72940b4d60 2024-03-25T18:27:57.899 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.898+0000 7f728a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:27:57.899 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.898+0000 7f728a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:27:57.900 INFO:teuthology.misc.health.smithi122.stdout: 2024-03-25T18:27:57.900 INFO:teuthology.misc.health.smithi122.stdout:{"epoch":11,"fsid":"b4b19611-696c-4b9f-a4ec-3ee2f6191d98","created":"2024-03-25T18:16:28.403512+0000","modified":"2024-03-25T18:26:32.649240+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:26:32.649240+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7303eaf0-b038-4c4f-94f6-0c358667adc1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"97ebd37c-5267-47ee-a84d-80a6e7444cde","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"188de57d-73fb-49be-84c4-92af3a291860","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"f1f49ae7-e393-4d82-9aa3-153bc6e286d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"eba478f5-2079-4357-8b03-5604ac7945c0","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"f8aa37f0-78b7-4515-952e-7e25f02c1e3b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"e36ed31b-7bed-4555-979c-e786ae626136","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"ad0b583f-133d-4389-80bd-293275a02d0f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:27:57.902 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.900+0000 7f729c093640 10 librados: watch_flush enter 2024-03-25T18:27:57.902 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.900+0000 7f729c093640 10 librados: watch_flush exit 2024-03-25T18:27:57.902 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.900+0000 7f729c093640 20 client.4706.objecter shutdown clearing up homeless session... 2024-03-25T18:27:57.902 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.900+0000 7f729c093640 10 client.4706.objecter successfully canceled tick 2024-03-25T18:27:57.903 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.900+0000 7f729c093640 1 -- 172.21.15.122:0/1076987834 >> v1:172.21.15.204:6800/676228005 conn(0x7f727806dd90 legacy=0x7f7278070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:57.903 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.901+0000 7f729c093640 10 monclient: shutdown 2024-03-25T18:27:57.903 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.901+0000 7f729c093640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:27:57.903 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.901+0000 7f729c093640 1 -- 172.21.15.122:0/1076987834 >> v1:172.21.15.122:6790/0 conn(0x7f72940b4d60 legacy=0x7f72941423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:27:57.903 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.901+0000 7f729c093640 1 -- 172.21.15.122:0/1076987834 shutdown_connections 2024-03-25T18:27:57.903 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.901+0000 7f729c093640 1 -- 172.21.15.122:0/1076987834 >> 172.21.15.122:0/1076987834 conn(0x7f72940617a0 msgr2=0x7f729413c780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:27:57.903 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.901+0000 7f729c093640 1 -- 172.21.15.122:0/1076987834 shutdown_connections 2024-03-25T18:27:57.903 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.901+0000 7f729c093640 1 -- 172.21.15.122:0/1076987834 wait complete. 2024-03-25T18:27:57.903 INFO:teuthology.misc.health.smithi122.stderr:2024-03-25T18:27:57.901+0000 7f729c093640 1 librados: shutdown 2024-03-25T18:27:57.914 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:27:57.914 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:27:57.916 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:27:57.916 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:27:57.916 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:27:58.047 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:11:51.789 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.052 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:11:51.794 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.096 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:11:51.838 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.102 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:11:51.844 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.106 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:11:51.852 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.108 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:11:51.850 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.123 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:11:51.868 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.133 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:11:51.876 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:58.146 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:11:51.892 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:27:59.919 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:11:25.606 35507== 2024-03-25T18:27:59.920 INFO:tasks.ceph.osd.0.smithi122.stderr:==00:00:11:25.606 35507== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:27:59.957 INFO:tasks.ceph.osd.0.smithi122.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:04.018 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:04.018 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi122 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:28:04.019 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:28:04.019 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:28:04.020 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:05.868 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:11:31.559 35506== 2024-03-25T18:28:05.869 INFO:tasks.ceph.osd.1.smithi122.stderr:==00:00:11:31.559 35506== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:05.904 INFO:tasks.ceph.osd.1.smithi122.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:10.121 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:10.122 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi122 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:28:10.122 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:28:10.123 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:28:10.123 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:12.182 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:11:37.881 35505== 2024-03-25T18:28:12.183 INFO:tasks.ceph.osd.2.smithi122.stderr:==00:00:11:37.881 35505== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:12.220 INFO:tasks.ceph.osd.2.smithi122.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:16.225 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:16.225 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi122 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:28:16.226 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:28:16.226 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:28:16.226 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:18.017 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:11:43.697 35508== 2024-03-25T18:28:18.017 INFO:tasks.ceph.osd.3.smithi122.stderr:==00:00:11:43.697 35508== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:18.032 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:12:11.774 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:18.059 INFO:tasks.ceph.osd.3.smithi122.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:18.086 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:12:11.828 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:18.107 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:12:11.849 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:18.131 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:12:11.877 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:18.136 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:12:11.882 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:18.136 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:12:11.879 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:18.140 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:12:11.883 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:18.152 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:12:11.898 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:18.169 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:12:11.911 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:22.328 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:22.328 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi122 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:28:22.329 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:28:22.329 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:28:22.329 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:24.405 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:11:50.108 35673== 2024-03-25T18:28:24.405 INFO:tasks.ceph.osd.4.smithi204.stderr:==00:00:11:50.108 35673== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:24.445 INFO:tasks.ceph.osd.4.smithi204.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:28.431 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:28.432 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi204 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:28:28.432 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:28:28.432 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:28:28.433 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:30.416 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:11:56.119 35674== 2024-03-25T18:28:30.416 INFO:tasks.ceph.osd.5.smithi204.stderr:==00:00:11:56.119 35674== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:30.457 INFO:tasks.ceph.osd.5.smithi204.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:34.534 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:34.535 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi204 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:28:34.535 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:28:34.536 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:28:34.536 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:36.241 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:12:01.940 35675== 2024-03-25T18:28:36.242 INFO:tasks.ceph.osd.6.smithi204.stderr:==00:00:12:01.940 35675== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:36.298 INFO:tasks.ceph.osd.6.smithi204.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:38.078 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:12:31.821 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:38.122 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:12:31.867 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:38.124 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:12:31.866 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:38.127 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:12:31.873 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:38.141 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:12:31.883 35193== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:38.145 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:12:31.890 35129== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:38.171 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:12:31.913 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:38.173 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:12:31.916 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:38.191 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:12:31.934 35195== Warning: unimplemented fcntl command: 1036 2024-03-25T18:28:40.637 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:40.637 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi204 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:28:40.638 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:28:40.638 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:28:40.638 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:42.190 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:12:07.887 35676== 2024-03-25T18:28:42.190 INFO:tasks.ceph.osd.7.smithi204.stderr:==00:00:12:07.887 35676== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:42.221 INFO:tasks.ceph.osd.7.smithi204.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:46.741 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:46.741 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi204 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:28:46.741 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:28:46.742 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:28:46.742 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:28:46.742 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:46.766 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:28:46.767 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:28:46.767 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:46.768 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:28:46.767+0000 7f4b75760640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 35127) UID: 0 2024-03-25T18:28:46.768 INFO:tasks.ceph.mgr.x.smithi204.stderr:2024-03-25T18:28:46.767+0000 7f4b75760640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:28:46.793 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:28:46.793 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:28:46.793 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:28:46.793 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:46.822 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-03-25T18:28:46.810+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35160) UID: 0 2024-03-25T18:28:46.822 INFO:tasks.ceph.mon.a.smithi122.stderr:2024-03-25T18:28:46.812+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:28:47.225 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:12:40.967 35193== 2024-03-25T18:28:47.225 INFO:tasks.ceph.mon.a.smithi122.stderr:==00:00:12:40.967 35193== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:47.234 INFO:tasks.ceph.mon.a.smithi122.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:52.895 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:52.895 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi122 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:28:52.895 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:28:52.896 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:28:52.896 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:52.911 INFO:tasks.ceph.mon.c.smithi122.stderr:2024-03-25T18:28:52.900+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35162) UID: 0 2024-03-25T18:28:52.912 INFO:tasks.ceph.mon.c.smithi122.stderr:2024-03-25T18:28:52.901+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:28:53.430 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:12:47.172 35195== 2024-03-25T18:28:53.430 INFO:tasks.ceph.mon.c.smithi122.stderr:==00:00:12:47.172 35195== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:53.440 INFO:tasks.ceph.mon.c.smithi122.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:28:58.998 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:28:58.998 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi122 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:28:58.999 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:28:58.999 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:28:58.999 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:28:59.015 INFO:tasks.ceph.mon.b.smithi204.stderr:2024-03-25T18:28:59.003+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 35125) UID: 0 2024-03-25T18:28:59.015 INFO:tasks.ceph.mon.b.smithi204.stderr:2024-03-25T18:28:59.004+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T18:28:59.423 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:12:53.168 35129== 2024-03-25T18:28:59.423 INFO:tasks.ceph.mon.b.smithi204.stderr:==00:00:12:53.168 35129== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:28:59.432 INFO:tasks.ceph.mon.b.smithi204.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:29:05.100 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:29:05.101 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi204 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:29:05.101 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:29:05.102 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:29:05.102 DEBUG:teuthology.orchestra.run.smithi122:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:29:05.211 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi122.front.sepia.ceph.com 2024-03-25T18:29:05.211 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:29:05.334 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi122.front.sepia.ceph.com 2024-03-25T18:29:05.334 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:29:05.406 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi122.front.sepia.ceph.com 2024-03-25T18:29:05.406 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:29:05.472 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi122.front.sepia.ceph.com 2024-03-25T18:29:05.472 DEBUG:teuthology.orchestra.run.smithi122:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:29:05.531 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-25T18:29:05.532 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:29:05.715 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-25T18:29:05.715 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:29:05.755 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-25T18:29:05.755 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:29:05.845 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi204.front.sepia.ceph.com 2024-03-25T18:29:05.845 DEBUG:teuthology.orchestra.run.smithi204:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:29:05.896 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:29:05.898 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621268/data/mon.a.tgz 2024-03-25T18:29:05.899 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-03-25T18:29:05.916 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.DMSGSktBBj 2024-03-25T18:29:05.916 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.DMSGSktBBj 2024-03-25T18:29:06.018 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.DMSGSktBBj 2024-03-25T18:29:06.107 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.DMSGSktBBj is 134KB 2024-03-25T18:29:06.160 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.DMSGSktBBj 2024-03-25T18:29:06.175 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621268/data/mon.c.tgz 2024-03-25T18:29:06.175 DEBUG:teuthology.orchestra.run.smithi122:> mktemp 2024-03-25T18:29:06.229 INFO:teuthology.orchestra.run.smithi122.stdout:/tmp/tmp.OFJNKZRkFF 2024-03-25T18:29:06.230 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.OFJNKZRkFF 2024-03-25T18:29:06.318 DEBUG:teuthology.orchestra.run.smithi122:> sudo chmod 0666 /tmp/tmp.OFJNKZRkFF 2024-03-25T18:29:06.402 DEBUG:teuthology.orchestra.remote:smithi122:/tmp/tmp.OFJNKZRkFF is 134KB 2024-03-25T18:29:06.424 DEBUG:teuthology.orchestra.run.smithi122:> rm -fr /tmp/tmp.OFJNKZRkFF 2024-03-25T18:29:06.438 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621268/data/mon.b.tgz 2024-03-25T18:29:06.438 DEBUG:teuthology.orchestra.run.smithi204:> mktemp 2024-03-25T18:29:06.455 INFO:teuthology.orchestra.run.smithi204.stdout:/tmp/tmp.cLIlBGbhNr 2024-03-25T18:29:06.455 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.cLIlBGbhNr 2024-03-25T18:29:06.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo chmod 0666 /tmp/tmp.cLIlBGbhNr 2024-03-25T18:29:06.650 DEBUG:teuthology.orchestra.remote:smithi204:/tmp/tmp.cLIlBGbhNr is 134KB 2024-03-25T18:29:06.702 DEBUG:teuthology.orchestra.run.smithi204:> rm -fr /tmp/tmp.cLIlBGbhNr 2024-03-25T18:29:06.716 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:29:06.717 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:29:06.719 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:29:07.104 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:29:07.376 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:29:07.378 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621268/remote/smithi122/crash 2024-03-25T18:29:07.379 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:29:07.406 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621268/remote/smithi204/crash 2024-03-25T18:29:07.407 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:29:07.433 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:29:07.434 DEBUG:teuthology.orchestra.run.smithi122:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:29:07.451 DEBUG:teuthology.orchestra.run.smithi204:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:29:07.525 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35283.log 2024-03-25T18:29:07.525 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:29:07.525 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35363.log 2024-03-25T18:29:07.525 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:29:07.525 INFO:teuthology.orchestra.run.smithi204.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.35283.log.gz 2024-03-25T18:29:07.525 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35363.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:29:07.525 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:29:07.525 INFO:teuthology.orchestra.run.smithi204.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.35363.log.gz 2024-03-25T18:29:07.526 INFO:teuthology.orchestra.run.smithi204.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:29:07.526 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:29:07.526 INFO:teuthology.orchestra.run.smithi204.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:29:07.526 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:29:07.526 INFO:teuthology.orchestra.run.smithi204.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:29:07.526 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35525.log 2024-03-25T18:29:07.526 INFO:teuthology.orchestra.run.smithi204.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:29:07.526 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35483.log 2024-03-25T18:29:07.526 INFO:teuthology.orchestra.run.smithi204.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:29:07.526 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35525.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:29:07.526 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35483.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:29:07.526 INFO:teuthology.orchestra.run.smithi204.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35525.log.gz 2024-03-25T18:29:07.526 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35323.log 2024-03-25T18:29:07.526 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35483.log.gz 2024-03-25T18:29:07.527 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35403.log 2024-03-25T18:29:07.527 INFO:teuthology.orchestra.run.smithi204.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:29:07.527 INFO:teuthology.orchestra.run.smithi204.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:29:07.527 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35323.log: /var/log/ceph/ceph-client.admin.35403.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:29:07.527 INFO:teuthology.orchestra.run.smithi204.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.35323.log.gz 2024-03-25T18:29:07.527 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.7.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35403.log.gzgzip 2024-03-25T18:29:07.527 INFO:teuthology.orchestra.run.smithi204.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35443.log 2024-03-25T18:29:07.527 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:29:07.527 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35574.log 2024-03-25T18:29:07.527 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:29:07.527 INFO:teuthology.orchestra.run.smithi204.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35443.log.gz 2024-03-25T18:29:07.527 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph-client.admin.35574.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:29:07.527 INFO:teuthology.orchestra.run.smithi204.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-mon.b.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35574.log.gz 2024-03-25T18:29:07.527 INFO:teuthology.orchestra.run.smithi204.stderr: 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:29:07.532 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40176.log 2024-03-25T18:29:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39077.log 2024-03-25T18:29:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29346.log 2024-03-25T18:29:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40176.log: /var/log/ceph/ceph-client.admin.39077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41971.log 2024-03-25T18:29:07.533 INFO:teuthology.orchestra.run.smithi122.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40176.log.gz 2024-03-25T18:29:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph.tmp-client.admin.29346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39705.log 0.0% 2024-03-25T18:29:07.533 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph.tmp-client.admin.29346.log.gz 2024-03-25T18:29:07.533 INFO:teuthology.orchestra.run.smithi122.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39077.log.gz 2024-03-25T18:29:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40098.log 2024-03-25T18:29:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39705.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:29:07.533 INFO:teuthology.orchestra.run.smithi122.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41971.log.gz 2024-03-25T18:29:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40098.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39471.log 2024-03-25T18:29:07.533 INFO:teuthology.orchestra.run.smithi122.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39705.log.gz 2024-03-25T18:29:07.533 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40762.log 2024-03-25T18:29:07.533 INFO:teuthology.orchestra.run.smithi122.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40098.log.gz 2024-03-25T18:29:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39471.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39861.log 2024-03-25T18:29:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40762.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40801.log 2024-03-25T18:29:07.534 INFO:teuthology.orchestra.run.smithi122.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39471.log.gz 2024-03-25T18:29:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39861.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40762.log.gz 2024-03-25T18:29:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41464.log 2024-03-25T18:29:07.534 INFO:teuthology.orchestra.run.smithi122.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39861.log.gz 2024-03-25T18:29:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42244.log 2024-03-25T18:29:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41464.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40216.log 2024-03-25T18:29:07.534 INFO:teuthology.orchestra.run.smithi122.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40801.log.gz 2024-03-25T18:29:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.42244.log: gzip -5 --verbose -- 86.3% /var/log/ceph/ceph-client.admin.39627.log 2024-03-25T18:29:07.534 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41464.log.gz 2024-03-25T18:29:07.534 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40216.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:29:07.534 INFO:teuthology.orchestra.run.smithi122.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42244.log.gz 2024-03-25T18:29:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39627.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:29:07.535 INFO:teuthology.orchestra.run.smithi122.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40216.log.gz 2024-03-25T18:29:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:29:07.535 INFO:teuthology.orchestra.run.smithi122.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39627.log.gz 2024-03-25T18:29:07.535 INFO:teuthology.orchestra.run.smithi122.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:29:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:29:07.535 INFO:teuthology.orchestra.run.smithi122.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:29:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:29:07.535 INFO:teuthology.orchestra.run.smithi122.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:29:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:29:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/valgrind/mon.a.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:29:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38837.log 2024-03-25T18:29:07.535 INFO:teuthology.orchestra.run.smithi122.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:29:07.535 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39276.log 2024-03-25T18:29:07.536 INFO:teuthology.orchestra.run.smithi122.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:29:07.536 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.38837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42088.log 2024-03-25T18:29:07.536 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40528.log 2024-03-25T18:29:07.536 INFO:teuthology.orchestra.run.smithi122.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38837.log.gz 2024-03-25T18:29:07.536 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.42088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41308.log 2024-03-25T18:29:07.536 INFO:teuthology.orchestra.run.smithi122.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39276.log.gz 2024-03-25T18:29:07.536 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40020.log 2024-03-25T18:29:07.536 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40528.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42088.log.gz 2024-03-25T18:29:07.536 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40996.log 2024-03-25T18:29:07.536 INFO:teuthology.orchestra.run.smithi122.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40528.log.gz 2024-03-25T18:29:07.536 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40020.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42166.log 2024-03-25T18:29:07.536 INFO:teuthology.orchestra.run.smithi122.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41308.log.gz 2024-03-25T18:29:07.536 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40996.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40020.log.gz 2024-03-25T18:29:07.536 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41932.log 2024-03-25T18:29:07.536 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.42166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39744.log 2024-03-25T18:29:07.537 INFO:teuthology.orchestra.run.smithi122.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40996.log.gz 2024-03-25T18:29:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41230.log 2024-03-25T18:29:07.537 INFO:teuthology.orchestra.run.smithi122.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42166.log.gz 2024-03-25T18:29:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39744.log: gzip -5 86.2% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.41932.log.gz 2024-03-25T18:29:07.537 INFO:teuthology.orchestra.run.smithi122.stderr: /var/log/ceph/ceph-client.admin.39354.log 2024-03-25T18:29:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39549.log 2024-03-25T18:29:07.537 INFO:teuthology.orchestra.run.smithi122.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39744.log.gz 2024-03-25T18:29:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39354.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41230.log.gz 2024-03-25T18:29:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40567.log 2024-03-25T18:29:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39549.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40918.log 2024-03-25T18:29:07.537 INFO:teuthology.orchestra.run.smithi122.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39354.log.gz 2024-03-25T18:29:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40567.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:29:07.537 INFO:teuthology.orchestra.run.smithi122.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39549.log.gz 2024-03-25T18:29:07.537 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41893.log 2024-03-25T18:29:07.537 INFO:teuthology.orchestra.run.smithi122.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40567.log.gz 2024-03-25T18:29:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40333.log 2024-03-25T18:29:07.538 INFO:teuthology.orchestra.run.smithi122.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40918.log.gz 2024-03-25T18:29:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41893.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42127.log 2024-03-25T18:29:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40333.log: 86.7% 93.4% -- replaced with /var/log/ceph/ceph-client.admin.41893.log.gz 2024-03-25T18:29:07.538 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:29:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39119.log 2024-03-25T18:29:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.42127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41347.log 2024-03-25T18:29:07.538 INFO:teuthology.orchestra.run.smithi122.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40333.log.gz 2024-03-25T18:29:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39038.log 2024-03-25T18:29:07.538 INFO:teuthology.orchestra.run.smithi122.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42127.log.gz 2024-03-25T18:29:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41347.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40684.log 2024-03-25T18:29:07.538 INFO:teuthology.orchestra.run.smithi122.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39119.log.gz 2024-03-25T18:29:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39588.log 2024-03-25T18:29:07.538 INFO:teuthology.orchestra.run.smithi122.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41347.log.gz 2024-03-25T18:29:07.538 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40684.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41269.log 2024-03-25T18:29:07.539 INFO:teuthology.orchestra.run.smithi122.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39038.log.gz 2024-03-25T18:29:07.539 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42010.log 2024-03-25T18:29:07.539 INFO:teuthology.orchestra.run.smithi122.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40684.log.gz 2024-03-25T18:29:07.539 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41269.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41542.log 2024-03-25T18:29:07.539 INFO:teuthology.orchestra.run.smithi122.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39588.log.gz 2024-03-25T18:29:07.539 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.42010.log: gzip 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41269.log.gz -5 2024-03-25T18:29:07.539 INFO:teuthology.orchestra.run.smithi122.stderr: --verbose -- /var/log/ceph/ceph-client.admin.36576.log 2024-03-25T18:29:07.539 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41542.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40059.log 2024-03-25T18:29:07.539 INFO:teuthology.orchestra.run.smithi122.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.42010.log.gz 2024-03-25T18:29:07.539 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.36576.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40840.log 2024-03-25T18:29:07.539 INFO:teuthology.orchestra.run.smithi122.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41542.log.gz 2024-03-25T18:29:07.539 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40059.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:29:07.539 INFO:teuthology.orchestra.run.smithi122.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.36576.log.gz 2024-03-25T18:29:07.539 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39510.log 2024-03-25T18:29:07.540 INFO:teuthology.orchestra.run.smithi122.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40059.log.gz 2024-03-25T18:29:07.540 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41659.log 2024-03-25T18:29:07.540 INFO:teuthology.orchestra.run.smithi122.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40840.log.gz 2024-03-25T18:29:07.540 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.39510.log: /var/log/ceph/ceph-client.admin.39237.log 2024-03-25T18:29:07.540 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42049.log 2024-03-25T18:29:07.540 INFO:teuthology.orchestra.run.smithi122.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39510.log.gz 2024-03-25T18:29:07.540 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35202.log 2024-03-25T18:29:07.540 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39237.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41659.log.gz 2024-03-25T18:29:07.540 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.42049.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35531.log 2024-03-25T18:29:07.540 INFO:teuthology.orchestra.run.smithi122.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39237.log.gz 2024-03-25T18:29:07.540 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35202.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42283.log 2024-03-25T18:29:07.540 INFO:teuthology.orchestra.run.smithi122.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42049.log.gz 2024-03-25T18:29:07.540 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40489.log 2024-03-25T18:29:07.540 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35531.log: /var/log/ceph/ceph-client.admin.42283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42322.log 2024-03-25T18:29:07.540 INFO:teuthology.orchestra.run.smithi122.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.35202.log.gz 2024-03-25T18:29:07.541 INFO:teuthology.orchestra.run.smithi122.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.35531.log.gz 2024-03-25T18:29:07.541 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39159.log 2024-03-25T18:29:07.541 INFO:teuthology.orchestra.run.smithi122.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42283.log.gz 2024-03-25T18:29:07.541 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.42322.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40489.log.gz 2024-03-25T18:29:07.541 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41698.log 2024-03-25T18:29:07.541 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39159.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39822.log 2024-03-25T18:29:07.541 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41698.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42322.log.gz 2024-03-25T18:29:07.541 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39783.log 2024-03-25T18:29:07.541 INFO:teuthology.orchestra.run.smithi122.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39159.log.gz 2024-03-25T18:29:07.541 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39822.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:29:07.541 INFO:teuthology.orchestra.run.smithi122.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41698.log.gz 2024-03-25T18:29:07.541 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42205.log 2024-03-25T18:29:07.541 INFO:teuthology.orchestra.run.smithi122.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39822.log.gz 2024-03-25T18:29:07.541 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.3.log: gzip 86.1% -5 -- replaced with /var/log/ceph/ceph-client.admin.39783.log.gz --verbose 2024-03-25T18:29:07.541 INFO:teuthology.orchestra.run.smithi122.stderr: -- /var/log/ceph/ceph-client.admin.41854.log 2024-03-25T18:29:07.541 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.42205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41113.log 2024-03-25T18:29:07.542 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41035.log 2024-03-25T18:29:07.542 INFO:teuthology.orchestra.run.smithi122.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.42205.log.gz 2024-03-25T18:29:07.542 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41113.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41854.log.gz 2024-03-25T18:29:07.542 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41620.log 2024-03-25T18:29:07.542 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39315.log 2024-03-25T18:29:07.542 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41035.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41113.log.gz 2024-03-25T18:29:07.542 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41503.log 2024-03-25T18:29:07.542 INFO:teuthology.orchestra.run.smithi122.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41035.log.gz 2024-03-25T18:29:07.542 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40606.log 2024-03-25T18:29:07.542 INFO:teuthology.orchestra.run.smithi122.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41620.log.gz 2024-03-25T18:29:07.542 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41503.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39315.log.gz 2024-03-25T18:29:07.542 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39981.log 2024-03-25T18:29:07.542 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40645.log 2024-03-25T18:29:07.542 INFO:teuthology.orchestra.run.smithi122.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41503.log.gz 2024-03-25T18:29:07.542 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39393.log 2024-03-25T18:29:07.543 INFO:teuthology.orchestra.run.smithi122.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40606.log.gz 2024-03-25T18:29:07.543 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40645.log: gzip -5 --verbose 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39981.log.gz 2024-03-25T18:29:07.543 INFO:teuthology.orchestra.run.smithi122.stderr: -- /var/log/ceph/ceph-client.admin.37658.log 2024-03-25T18:29:07.543 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39903.log 2024-03-25T18:29:07.543 INFO:teuthology.orchestra.run.smithi122.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40645.log.gz 2024-03-25T18:29:07.543 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.37658.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40879.log 2024-03-25T18:29:07.543 INFO:teuthology.orchestra.run.smithi122.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39393.log.gz 2024-03-25T18:29:07.543 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41191.log 2024-03-25T18:29:07.543 INFO:teuthology.orchestra.run.smithi122.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.37658.log.gz 2024-03-25T18:29:07.543 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40879.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:29:07.543 INFO:teuthology.orchestra.run.smithi122.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39903.log.gz 2024-03-25T18:29:07.543 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41191.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41386.log 2024-03-25T18:29:07.543 INFO:teuthology.orchestra.run.smithi122.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40879.log.gz 2024-03-25T18:29:07.543 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41815.log 2024-03-25T18:29:07.543 INFO:teuthology.orchestra.run.smithi122.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41191.log.gz 2024-03-25T18:29:07.544 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41386.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40450.log 2024-03-25T18:29:07.544 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41815.log: 86.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41581.log 2024-03-25T18:29:07.544 INFO:teuthology.orchestra.run.smithi122.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41386.log.gz 2024-03-25T18:29:07.544 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41776.log 2024-03-25T18:29:07.544 INFO:teuthology.orchestra.run.smithi122.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41815.log.gz 2024-03-25T18:29:07.544 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35504.log 2024-03-25T18:29:07.544 INFO:teuthology.orchestra.run.smithi122.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40450.log.gz/var/log/ceph/ceph-client.admin.41581.log: 2024-03-25T18:29:07.544 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41776.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41074.log 2024-03-25T18:29:07.544 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35504.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41581.log.gz 87.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.41776.log.gz -5 2024-03-25T18:29:07.544 INFO:teuthology.orchestra.run.smithi122.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41737.log 2024-03-25T18:29:07.544 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-25T18:29:07.544 INFO:teuthology.orchestra.run.smithi122.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.35504.log.gz 2024-03-25T18:29:07.544 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41074.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41425.log 2024-03-25T18:29:07.544 INFO:teuthology.orchestra.run.smithi122.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41074.log.gz 2024-03-25T18:29:07.544 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41737.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39666.log 2024-03-25T18:29:07.544 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41425.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:29:07.545 INFO:teuthology.orchestra.run.smithi122.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41737.log.gz 2024-03-25T18:29:07.545 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39666.log: gzip 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41425.log.gz 2024-03-25T18:29:07.545 INFO:teuthology.orchestra.run.smithi122.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40723.log 2024-03-25T18:29:07.545 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.0.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39666.log.gz 2024-03-25T18:29:07.545 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40255.log 2024-03-25T18:29:07.545 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40723.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39942.log 2024-03-25T18:29:07.545 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40255.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40723.log.gz 2024-03-25T18:29:07.545 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:29:07.545 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40255.log.gz 2024-03-25T18:29:07.545 INFO:teuthology.orchestra.run.smithi122.stderr: --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:29:07.545 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph.log: /var/log/ceph/ceph-client.admin.39942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41152.log 2024-03-25T18:29:07.545 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-osd.2.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39942.log.gz 2024-03-25T18:29:07.545 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40411.log 2024-03-25T18:29:07.545 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.41152.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:29:07.545 INFO:teuthology.orchestra.run.smithi122.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41152.log.gz 2024-03-25T18:29:07.546 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39432.log 2024-03-25T18:29:07.546 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40957.log 2024-03-25T18:29:07.546 INFO:teuthology.orchestra.run.smithi122.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40411.log.gz 2024-03-25T18:29:07.546 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.39432.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40294.log 2024-03-25T18:29:07.546 INFO:teuthology.orchestra.run.smithi122.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.39432.log.gz 2024-03-25T18:29:07.546 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40957.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40137.log 2024-03-25T18:29:07.546 INFO:teuthology.orchestra.run.smithi122.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40957.log.gz/var/log/ceph/ceph-client.admin.40294.log: 2024-03-25T18:29:07.546 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39198.log 2024-03-25T18:29:07.546 INFO:teuthology.orchestra.run.smithi122.stderr: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:29:07.546 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40137.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40294.log.gz 2024-03-25T18:29:07.546 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40372.log 2024-03-25T18:29:07.546 INFO:teuthology.orchestra.run.smithi122.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40137.log.gz/var/log/ceph/ceph-client.admin.39198.log: 2024-03-25T18:29:07.546 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35684.log 2024-03-25T18:29:07.546 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.40372.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39198.log.gz 2024-03-25T18:29:07.546 INFO:teuthology.orchestra.run.smithi122.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:29:07.547 INFO:teuthology.orchestra.run.smithi122.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40372.log.gz 2024-03-25T18:29:07.547 INFO:teuthology.orchestra.run.smithi122.stderr:/var/log/ceph/ceph-client.admin.35684.log: /var/log/ceph/ceph-mon.c.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35684.log.gz 2024-03-25T18:29:07.588 INFO:teuthology.orchestra.run.smithi204.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:29:08.357 INFO:teuthology.orchestra.run.smithi122.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:29:08.401 INFO:teuthology.orchestra.run.smithi204.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:29:08.410 INFO:teuthology.orchestra.run.smithi204.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:29:08.618 INFO:teuthology.orchestra.run.smithi122.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:29:08.697 INFO:teuthology.orchestra.run.smithi204.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:29:08.706 INFO:teuthology.orchestra.run.smithi122.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:29:08.722 INFO:teuthology.orchestra.run.smithi122.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:29:08.746 INFO:teuthology.orchestra.run.smithi204.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:29:12.354 INFO:teuthology.orchestra.run.smithi204.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:29:12.356 INFO:teuthology.orchestra.run.smithi204.stderr: 2024-03-25T18:29:12.356 INFO:teuthology.orchestra.run.smithi204.stderr:real 0m4.862s 2024-03-25T18:29:12.356 INFO:teuthology.orchestra.run.smithi204.stderr:user 0m8.840s 2024-03-25T18:29:12.356 INFO:teuthology.orchestra.run.smithi204.stderr:sys 0m0.326s 2024-03-25T18:29:12.647 INFO:teuthology.orchestra.run.smithi122.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:29:12.647 INFO:teuthology.orchestra.run.smithi122.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:29:12.649 INFO:teuthology.orchestra.run.smithi122.stderr: 2024-03-25T18:29:12.649 INFO:teuthology.orchestra.run.smithi122.stderr:real 0m5.187s 2024-03-25T18:29:12.649 INFO:teuthology.orchestra.run.smithi122.stderr:user 0m14.306s 2024-03-25T18:29:12.649 INFO:teuthology.orchestra.run.smithi122.stderr:sys 0m0.531s 2024-03-25T18:29:12.649 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:29:12.650 DEBUG:teuthology.misc:Transferring archived files from smithi122:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621268/remote/smithi122/log 2024-03-25T18:29:12.651 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:29:15.285 DEBUG:teuthology.misc:Transferring archived files from smithi204:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621268/remote/smithi204/log 2024-03-25T18:29:15.286 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:29:16.834 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:29:17.042 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=df27734a00674be9917173d0b8b41406 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:29:17.046 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:29:17.056 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:29:17.100 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:29:17.100 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:29:17.117 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:29:17.196 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:29:17.205 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:29:17.205 DEBUG:teuthology.orchestra.run.smithi122:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:29:17.208 DEBUG:teuthology.orchestra.run.smithi204:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:29:17.221 INFO:teuthology.orchestra.run.smithi122.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:29:17.224 INFO:teuthology.orchestra.run.smithi204.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:29:17.225 INFO:teuthology.orchestra.run.smithi122.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:29:17.226 INFO:teuthology.orchestra.run.smithi122.stdout:=============================================================================== 2024-03-25T18:29:17.226 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 35 +588us[ +588us] +/- 68ms 2024-03-25T18:29:17.226 INFO:teuthology.orchestra.run.smithi122.stdout:^* hv02.front.sepia.ceph.com 3 8 377 232 -1204us[-1351us] +/- 50ms 2024-03-25T18:29:17.226 INFO:teuthology.orchestra.run.smithi122.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +682us[ +682us] +/- 73ms 2024-03-25T18:29:17.226 INFO:teuthology.orchestra.run.smithi122.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:17.229 INFO:teuthology.orchestra.run.smithi204.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:29:17.229 INFO:teuthology.orchestra.run.smithi204.stdout:=============================================================================== 2024-03-25T18:29:17.229 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 35 +815us[ +815us] +/- 68ms 2024-03-25T18:29:17.229 INFO:teuthology.orchestra.run.smithi204.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -1075us[-1102us] +/- 53ms 2024-03-25T18:29:17.229 INFO:teuthology.orchestra.run.smithi204.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 +900us[ +873us] +/- 73ms 2024-03-25T18:29:17.229 INFO:teuthology.orchestra.run.smithi204.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:29:17.230 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:29:17.238 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:29:17.239 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:29:17.267 DEBUG:teuthology.orchestra.run.smithi122:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:29:17.337 DEBUG:teuthology.orchestra.run.smithi204:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:29:17.409 DEBUG:teuthology.orchestra.run.smithi122:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:29:17.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:29:17.440 DEBUG:teuthology.orchestra.run.smithi204:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:29:17.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:29:17.472 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:29:17.482 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:29:17.526 INFO:teuthology.task.internal:Duration was 7681.431278 seconds 2024-03-25T18:29:17.526 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:29:17.560 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:29:17.560 DEBUG:teuthology.orchestra.run.smithi122:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:29:17.562 DEBUG:teuthology.orchestra.run.smithi204:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:29:17.604 INFO:teuthology.orchestra.run.smithi122.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:29:17.609 INFO:teuthology.orchestra.run.smithi204.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:29:17.827 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:29:17.828 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi122.front.sepia.ceph.com 2024-03-25T18:29:17.828 DEBUG:teuthology.orchestra.run.smithi122:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:29:17.887 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi204.front.sepia.ceph.com 2024-03-25T18:29:17.887 DEBUG:teuthology.orchestra.run.smithi204:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:29:17.911 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:29:17.911 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:29:17.929 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:29:17.980 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:29:17.980 DEBUG:teuthology.orchestra.run.smithi122:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:29:17.998 DEBUG:teuthology.orchestra.run.smithi204:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:29:18.126 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:29:18.135 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:29:18.136 DEBUG:teuthology.orchestra.run.smithi122:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:29:18.163 DEBUG:teuthology.orchestra.run.smithi204:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:29:18.190 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:29:18.200 DEBUG:teuthology.orchestra.run.smithi122:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:29:18.206 DEBUG:teuthology.orchestra.run.smithi204:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:29:18.229 INFO:teuthology.orchestra.run.smithi122.stdout:kernel.core_pattern = core 2024-03-25T18:29:18.256 INFO:teuthology.orchestra.run.smithi204.stdout:kernel.core_pattern = core 2024-03-25T18:29:18.275 DEBUG:teuthology.orchestra.run.smithi122:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:29:18.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:29:18.316 DEBUG:teuthology.orchestra.run.smithi204:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:29:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:29:18.330 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:29:18.339 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:29:18.340 DEBUG:teuthology.misc:Transferring archived files from smithi122:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621268/remote/smithi122 2024-03-25T18:29:18.340 DEBUG:teuthology.orchestra.run.smithi122:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:29:18.391 DEBUG:teuthology.misc:Transferring archived files from smithi204:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621268/remote/smithi204 2024-03-25T18:29:18.392 DEBUG:teuthology.orchestra.run.smithi204:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:29:18.429 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:29:18.430 DEBUG:teuthology.orchestra.run.smithi122:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:29:18.432 DEBUG:teuthology.orchestra.run.smithi204:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:29:18.479 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:29:18.491 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:29:18.492 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:29:18.511 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:29:18.511 DEBUG:teuthology.orchestra.run.smithi122:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:29:18.513 DEBUG:teuthology.orchestra.run.smithi204:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:29:18.525 INFO:teuthology.orchestra.run.smithi122.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:29 /home/ubuntu/cephtest 2024-03-25T18:29:18.533 INFO:teuthology.orchestra.run.smithi204.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:29 /home/ubuntu/cephtest 2024-03-25T18:29:18.534 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:29:18.543 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:29:18.611 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:29:18.638 DEBUG:teuthology.nuke:shortname: smithi122 2024-03-25T18:29:18.639 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:29:18.666 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi122.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621268', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:11:41.766133', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGd7eAzSBHm5d/7BTcOsLLUMmH7tiZ52/vuXEdLczaKP5XPiYpF/sMjLY0JGojUMgBC7U39UxBfDgCB94CgY6dc='} 2024-03-25T18:29:18.670 DEBUG:teuthology.nuke:shortname: smithi204 2024-03-25T18:29:18.671 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:29:18.692 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi204.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621268', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:11:41.767667', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEF5xawqKZ/fdvHRzou19KJf/TWLawveGfsRh6QIGO3NEYljtxhlVih68qNJP/Tfjrq49W/+x4/ZkKNnGSgRMmA='} 2024-03-25T18:29:18.726 INFO:teuthology.orchestra.console.smithi122:Power off 2024-03-25T18:29:18.726 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:29:18.752 DEBUG:teuthology.orchestra.console.smithi122:power off output: Chassis Power Control: Down/Off 2024-03-25T18:29:18.753 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:29:18.769 INFO:teuthology.orchestra.console.smithi204:Power off 2024-03-25T18:29:18.770 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:29:18.779 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-03-25T18:29:18.792 DEBUG:teuthology.orchestra.console.smithi204:power off output: Chassis Power Control: Down/Off 2024-03-25T18:29:18.792 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:29:18.815 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-03-25T18:29:22.780 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:29:22.816 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:29:22.890 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-03-25T18:29:22.926 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-03-25T18:29:26.895 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:29:26.927 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:29:27.005 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is on 2024-03-25T18:29:27.040 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is on 2024-03-25T18:29:31.006 DEBUG:teuthology.orchestra.console.smithi122:pexpect command: ipmitool -H smithi122.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:29:31.040 DEBUG:teuthology.orchestra.console.smithi204:pexpect command: ipmitool -H smithi204.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:29:31.116 DEBUG:teuthology.orchestra.console.smithi122:check power output: Chassis Power is off 2024-03-25T18:29:31.150 DEBUG:teuthology.orchestra.console.smithi204:check power output: Chassis Power is off 2024-03-25T18:29:31.218 INFO:teuthology.orchestra.console.smithi122:Power off completed 2024-03-25T18:29:31.251 INFO:teuthology.orchestra.console.smithi204:Power off completed 2024-03-25T18:29:31.357 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7681.43127822876 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=df27734a00674be9917173d0b8b41406 status: fail success: false 2024-03-25T18:29:31.358 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:29:31.444 INFO:teuthology.run:FAIL